A Practical Guide to UNIX® for Mac OS® X Users

Next Page

Book Cover
A Practical Guide to UNIX® for Mac OS® X Users
By Mark G. Sobell, Peter Seebach
...............................................
Publisher: Prentice Hall
Pub Date: December 21, 2005
Print ISBN-10: 0-13-186333-9
Print ISBN-13: 978-0-13-186333-0
Pages: 1056

Table of Contents | Index
Overview

The Most Useful UNIX Guide for Mac OS X Users Ever, with Hundreds of High-Quality Examples!

Beneath Mac OS® X's stunning graphical user interface (GUI) is the most powerful operating system ever created: UNIX®. With unmatched clarity and insight, this book explains UNIX for the Mac OS X usergiving you total control over your system, so you can get more done, faster. Building on Mark Sobell's highly praised A Practical Guide to the UNIX System, it delivers comprehensive guidance on the UNIX command line tools every user, administrator, and developer needs to mastertogether with the world's best day-to-day UNIX reference.

This book is packed with hundreds of high-quality examples. From networking and system utilities to shells and programming, this is UNIX from the ground upboth the "whys" and the "hows"for every Mac user. You'll understand the relationships between GUI tools and their command line counterparts. Need instant answers? Don't bother with confusing online "manual pages": rely on this book's example-rich, quick-access, 236-page command reference!

Next Page


Table of Contents

Previous Page
Next Page

Book Cover
A Practical Guide to UNIX® for Mac OS® X Users
By Mark G. Sobell, Peter Seebach
...............................................
Publisher: Prentice Hall
Pub Date: December 21, 2005
Print ISBN-10: 0-13-186333-9
Print ISBN-13: 978-0-13-186333-0
Pages: 1056

Table of Contents | Index

Copyright
Praise for Mark Sobell's Books
Preface
 | Features of This Book
 | Contents
 | Supplements
 | Thanks from Mark G. Sobell
Chapter 1. Welcome to Mac OS X
 | The Heritage of Mac OS X: UNIX
 | The Open Source Connection
 | Overview of Mac OS X
 | Additional Features of Mac OS X
 | Chapter Summary
 | Exercises
Part I: The Mac OS X Operating System
 | Chapter 2. Getting Started
 |  | Conventions Used in This Book
 |  | Logging In
 |  | Curbing Your Power: Superuser/root/Administrator Access
 |  | Working with the Shell
 |  | Useful Graphical Tools
 |  | Getting the Facts: Where to Find Documentation
 |  | More About Logging In
 |  | Chapter Summary
 |  | Exercises
 |  | Advanced Exercises
 | Chapter 3. The Mac OS X Utilities
 |  | Special Characters
 |  | Basic Utilities
 |  | Working with Files
 |  | Three More Utilities
 |  | Compressing and Archiving Files
 |  | Locating Commands
 |  | Obtaining User and System Information
 |  | Communicating with Other Users
 |  | Email
 |  | Chapter Summary
 |  | Exercises
 |  | Advanced Exercises
 | Chapter 4. The Mac OS X Filesystem
 |  | The Hierarchical Filesystem
 |  | Directory Files and Ordinary Files
 |  | Pathnames
 |  | Directory Commands
 |  | Filesystems
 |  | Access Permissions
 |  | Extended Attributes
 |  | Types of Files
 |  | Links
 |  | Chapter Summary
 |  | Exercises
 |  | Advanced Exercises
 | Chapter 5. The Shell
 |  | The Command Line
 |  | Standard Input and Standard Output
 |  | Running a Program in the Background
 |  | Filename Generation/Pathname Expansion
 |  | Builtins
 |  | Chapter Summary
 |  | Exercises
 |  | Advanced Exercises
Part II: The Editors
 | Chapter 6. The vim Editor
 |  | History
 |  | Tutorial: Creating and Editing a File with vim
 |  | The compatible Parameter
 |  | Introduction to vim Features
 |  | Command Mode: Moving the Cursor
 |  | Input Mode
 |  | Command Mode: Deleting and Changing Text
 |  | Searching and Substituting
 |  | Miscellaneous Commands
 |  | Yank, Put, and Delete Commands
 |  | Reading and Writing Files
 |  | Setting Parameters
 |  | Advanced Editing Techniques
 |  | Units of Measure
 |  | Chapter Summary
 |  | Exercises
 |  | Advanced Exercises
 | Chapter 7. The emacs Editor
 |  | History
 |  | Tutorial: Getting Started with emacs
 |  | Basic Editing Commands
 |  | Online Help
 |  | Advanced Editing
 |  | Language-Sensitive Editing
 |  | More Information
 |  | Chapter Summary
 |  | Exercises
 |  | Advanced Exercises
Part III: The Shells
 | Chapter 8. The Bourne Again Shell
 |  | Background
 |  | Shell Basics
 |  | Parameters and Variables
 |  | Processes
 |  | History
 |  | Aliases
 |  | Functions
 |  | Controlling bash Features and Options
 |  | Processing the Command Line
 |  | Chapter Summary
 |  | Exercises
 |  | Advanced Exercises
 | Chapter 9. The TC Shell
 |  | Shell Scripts
 |  | Entering and Leaving the TC Shell
 |  | Features Common to the Bourne Again and TC Shells
 |  | Redirecting Standard Error
 |  | Working with the Command Line
 |  | Variables
 |  | Control Structures
 |  | Builtins
 |  | Chapter Summary
 |  | Exercises
 |  | Advanced Exercises
Part IV: Networking and Maintenance
 | Chapter 10. Networking and the Internet
 |  | Types of Networks and How They Work
 |  | Communicate over a Network
 |  | Network Utilities
 |  | Distributed Computing
 |  | Usenet
 |  | WWW: World Wide Web
 |  | Chapter Summary
 |  | Exercises
 |  | Advanced Exercises
 | Chapter 11. System Maintenance
 |  | Philosophy of System Administration
 |  | System Administrator and Superuser
 |  | System Operation
 |  | NetInfo
 |  | Important Standard Directories and Files
 |  | Useful Utilities
 |  | Setting Up a Server
 |  | PAM
 |  | fink: Downloads and Installs Software
 |  | Chapter Summary
 |  | Exercises
 |  | Advanced Exercises
Part V: Programming Tools
 | Chapter 12. Programming Tools
 |  | Carbon, Cocoa, and UNIX APIs
 |  | Programming in C
 |  | Using Shared Libraries
 |  | make: Keeps a Set of Programs Current
 |  | Debugging C Programs
 |  | Threads
 |  | System Calls
 |  | Source Code Management
 |  | Chapter Summary
 |  | Exercises
 |  | Advanced Exercises
 | Chapter 13. Programming the Bourne Again Shell
 |  | Control Structures
 |  | File Descriptors
 | Chapter 14. The awk Pattern Processing Language
 |  | Syntax
 |  | Arguments
 |  | Options
 |  | Notes
 |  | Language Basics
 |  | Examples
 |  | Error Messages
 |  | Chapter Summary
 |  | Exercises
 |  | Advanced Exercises
 | Chapter 15. The sed Editor
 |  | Syntax
 |  | Arguments
 |  | Options
 |  | Editor Basics
 |  | Examples
 |  | Chapter Summary
 |  | Exercises
Part VI: Command Reference
 | Command Reference
 |  | Utilities That Display and Manipulate Files
 |  | Network Utilities
 |  | Utilities That Display and Alter Status
 |  | Utilities That Are Programming Tools
 |  | Miscellaneous Utilities
 |  | Standard Multiplicative Suffixes
 |  | Common Options
 |  | The sample Utility
Part VII: Appendixes
 | Appendix A. Regular Expressions
 |  | Characters
 |  | Delimiters
 |  | Simple Strings
 |  | Special Characters
 |  | Rules
 |  | Bracketing Expressions
 |  | The Replacement String
 |  | Extended Regular Expressions
 |  | Appendix Summary
 | Appendix B. Help
 |  | Solving a Problem
 |  | Finding Related Information
 |  | Specifying a Terminal
 | Appendix C. Mac OS X for UNIX Users
 | Glossary
Index

Previous Page


Copyright

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals.

The authors and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein.

The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales, which may include electronic versions and/or custom covers and content particular to your business, training goals, marketing focus, and branding interests. For more information, please contact:

U.S. Corporate and Government Sales
(800) 382-3419
corpsales@pearsontechgroup.com

For sales outside the U.S., please contact:

International Sales
international@pearsoned.com

Visit us on the Web: www.prenhallprofessional.com

Library of Congress Cataloging-in-Publication Data

Sobell, Mark G.
A practical guide to Unix for Mac OS X users / Mark G. Sobell, Peter Seebach.
p. cm.
Includes index.
ISBN 0-13-186333-9 (pbk. : alk. paper)
1. UNIX (Computer file) 2. Mac OS. 3. Operating systems (Computers) I. Seebach, Peter. II. Title.
QA76.76.O63S597 2005
005.4'46dc22
2005030751

Copyright © 2006 Mark G. Sobell

All rights reserved. Printed in the United States of America. This publication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. For information regarding permissions, write to:

Pearson Education, Inc.
Rights and Contracts Department
One Lake Street
Upper Saddle River, NJ 07458

Text printed in the United States on recycled paper at Courier in Stoughton, Massachusetts.

First printing, December 2005

Dedication

For my uncle, David Z. Levitov (1920-2005), who gave me the world. M.G.S.


Praise for Mark Sobell's Books

"If you want to become a real Linux guru, this is one of the better books available. Readable, straightforward, educational, it is a superb reference that blends the educational aspect of a typical book on learning Linux with a typical book of command line references. I highly recommend A Practical Guide to Linux® Commands, Editors, and Shell Programming."

Harold D. McFarland Editor, Readers Preference Reviews

"I keep searching for books that collect everything you want to know about a subject in one place, and keep getting disappointed. Usually the books leave out some important topic, while others go too deep in some areas and must skim lightly over the others. A Practical Guide to Red Hat® Linux® is one of those rare books that actually pulls it off. Mark G. Sobell has created a single reference for Red Hat Linux that cannot be beat! This marvelous text (with a 4-CD set of Linux Fedora Core 2 included) is well worth the price. This is as close to an "everything you ever needed to know" book that I've seen. It's just that good and rates 5 out of 5."

Ray Lodato Slashdot contributor

"Mark Sobell has written a book as approachable as it is authoritative."

Jeffrey Bianchine Advocate, Author, Journalist

"Excellent reference book, well suited for the sysadmin of a Linux cluster, or the owner of a PC contemplating installing a recent stable Linux. Don't be put off by the daunting heft of the book. Sobell has striven to be as inclusive as possible, in trying to anticipate your system administration needs."

Wes Boudville Inventor

"A Practical Guide to Red Hat® Linux® is a brilliant book. Thank you Mark Sobell."

C. Pozrikidis University of California at San Diego

"This book presents the best overview of the Linux operating system that I have found. . . . [It] should be very helpful and understandable no matter what the reader's background is: traditional UNIX user, new Linux devotee, or even Windows user. Each topic is presented in a clear, complete fashion and very few assumptions are made about what the reader knows. . . . The book is extremely useful as a reference, as it contains a 70-page glossary of terms and is very well indexed. It is organized in such a way that the reader can focus on simple tasks without having to wade through more advanced topics until they are ready."

Cam Marshall Marshall Information Service LLC Member of Front Range UNIX Users Group [FRUUG] Boulder, Colorado

"Conclusively, this is THE book to get if you are a new Linux user and you just got into the RH/Fedora world. There's no other book that discusses so many different topics and in such depth."

Eugenia Loli-Queru Editor in Chief OSNews.com


Preface

A Practical Guide to UNIX® for Mac OS® X Users explains how to work with the UNIX operating system that is the foundation of Mac OS X. This book looks "under the hood," past the traditional graphical user interface (GUI) that most people think of as a Macintosh, and explains how to use the powerful command line interface (CLI) that connects you directly to UNIX.

Command line interface (CLI)

In the beginning UNIX had a command line (textual) interface. There was no mouse to point or icons to drag and drop. Some programs, such as emacs, implemented rudimentary windows using the very minimal graphics available in the ASCII character set. In addition, reverse video helped separate areas of the screen. UNIX was born and raised in this environment.

Naturally, all of the original UNIX tools were invoked from the command line. The real power of UNIX, and of Mac OS X, lies in this environment, which explains why many UNIX professionals work exclusively from the command line.

Using clear descriptions and lots of examples, this book shows you how to get the most out of your UNIX-based Mac OS X system using the command line interface. The first few chapters quickly bring readers with little computer experience up to speed. The rest of the book is appropriate for more experienced computer users.

Audience

This book is designed for a wide range of readers. It does not require programming experience, but assumes a basic familiarity with the Macintosh GUI. It is appropriate for the following readers:

  • Beginning Macintosh users who want to know what UNIX is, why everyone keeps saying that it is important, and how to take advantage of it

  • Experienced Macintosh users who want to know how to take advantage of the power of UNIX that underlies Mac OS X

  • Students who are taking a class in which they use Mac OS X

  • Power users who want to explore the power of Mac OS X from the command line

  • Professionals who use Mac OS X at work

  • UNIX users who want to adapt their UNIX skills to the Mac OS X environment

  • Computer science students who are studying the Mac OS X operating system

  • Programmers who need to understand the Mac OS X programming environment

  • Technical executives who want to get a grounding in Mac OS X

Benefits

A Practical Guide to UNIX® for Mac OS® X Users gives you an in-depth understanding of how to use the UNIX operating system that is the foundation for Mac OS X.

A large amount of free software has always been available for Macintosh systems. In addition, the Macintosh shareware community is very active. By introducing the UNIX aspects of Mac OS X, this book throws open to Macintosh users the vast store of free and low-cost software available for UNIX, Linux, and other UNIX-like systems.

Regardless of your background, this book offers the practical knowledge you need to get on with your work. You will come away from this book understanding how to use the UNIX operating system that underlies OS X, and this text will remain a valuable reference for years to come.


Features of This Book

This book is organized for ease of use in different situations. For example, you can read it from cover to cover to learn about the UNIX aspects of Mac OS X from the ground up. Alternatively, once you are comfortable using OS X, you can use this book as a reference: Look up a topic of interest in the table of contents or index and read about it. Or, refer to one of the utilities covered in Part VI, "Command Reference." You can also think of this book as a catalog of Mac OS X topics: Flip through the pages until a topic catches your eye. If you are familiar with UNIX or a UNIX-like operating system such as Linux, refer to Appendix C, "Mac OS X for UNIX Users," which lists some of the differences between Mac OS X and traditional UNIX systems. The book also includes many pointers to Web sites where you can get additional information: Consider the Web to be an extension of this book.

A Practical Guide to UNIX® for Mac OS® X Users offers these features:

  • Optional sections allow you to read the book at different levels, returning to more difficult material when you are ready to tackle it.

  • Caution boxes highlight procedures that can easily go wrong, giving you guidance before you run into trouble.

  • Tip boxes highlight situations in which you can save time by doing something differently or when it may be useful or just interesting to have additional information.

  • Security boxes point out ways that you can make a system more secure.

  • The supporting Web site at www.sobell.com includes corrections to the book, downloadable examples from the book, pointers to useful Web sites, and answers to even-numbered exercises.

  • Important command line utilities that were developed by Apple specifically for Mac OS X are covered in detail, including GetFileInfo, SetFile, nidump, otool, launchctl, diskutil, and plutil.

  • Descriptions of Mac OS X extended attributes include file forks, file attributes, attribute flags, and Access Control Lists (ACLs).

  • The relationships between GUI tools and their CLI counterparts are discussed in depth.

  • Information that will help you set up servers includes sections on property lists, the launchd superserver, and DHCP.

  • A section on NetInfo discusses the NetInfo database and ways to work with it.

  • Concepts are illustrated by practical examples found throughout the book.

  • Many useful URLs (Internet addresses) identify Web sites where you can obtain software and information.

  • Chapter Summaries review the important points covered in each chapter.

  • Review exercises at the end of each chapter allow readers to hone their skills. Answers to even-numbered exercises are available at www.sobell.com.

  • Important GNU tools, including gcc, gdb, GNU Configure and Build System, make, gzip, and many others, are described in detail.

  • Pointers throughout the book provide help in obtaining online documentation from many sources, including the local system and the Internet.


Contents

This section describes the information that each chapter covers and explains how that information can help you take advantage of the power of Mac OS X. You may want to review the table of contents for more detail.

  • Chapter 1 Welcome to Mac OS X

    Presents background information on Mac OS X. This chapter covers the history of Mac OS X, explains the connection between OS X and open-source software including GNU and BSD software, and discusses some of OS X's important features that distinguish it from other operating systems, including other versions of UNIX.

Part I: The Mac OS X Operating System

Tip: Experienced users may want to skim Part I

If you have used a UNIX-like system before, you may want to skim or skip some or all of the chapters in Part I. All readers should take a look at "Conventions Used in This Book" (page 18), which explains the typographic conventions that this book uses, and "Getting the Facts: Where to Find Documentation" (page 27), which points you toward both local and remote sources of Mac OS X documentation.

Part I introduces UNIX on a Mac OS X system and gets you started using it from the command line.

  • Chapter 2 Getting Started

    Explains the typographic conventions this book uses to make explanations clearer and easier to read. This chapter provides basic information and explains how to log in, display a shell prompt, give OS X commands using the shell, and find system documentation.

  • Chapter 3 The Mac OS X Utilities

    Explains the command line interface (CLI) and briefly introduces more than 30 command line utilities. Working through this chapter gives you a feel for UNIX and introduces some of the tools you will use day in and day out. Chapter 3 also introduces pipes, which allow you to combine utilities on the command line. The utilities covered in this chapter include

    • grep, which searches through files for strings of characters;

    • ditto, which copies files and directories (folders);

    • tar, which creates archive files that can hold many other files;

    • bzip2 and gzip, which compress files so that they take up less space on disk and allow you to transfer them over a network more quickly; and

    • diff, which displays the differences between two text files.

  • Chapter 4 The Mac OS X Filesystem

    Discusses the OS X hierarchical filesystem from a UNIX perspective, covering files, filenames, pathnames, working with directories (folders), access permissions, and hard and symbolic links. This chapter also discusses extended attributes including file forks, attribute flags, type codes, creator codes, and Access Control Lists (ACLs). Understanding the filesystem allows you to organize your data so that you can find information quickly. It also enables you to share some of your files with other users while keeping other files private.

  • Chapter 5 The Shell

    Explains how to use shell features to make your work faster and easier. All of the features covered in this chapter work with both the Bourne Again Shell (bash) and the TC Shell (tcsh). This chapter discusses

    • Using command line options to modify the way a command works;

    • How a minor change in a command line can redirect input to a command to come from a file instead of from the keyboard;

    • How to redirect output from a command to go to a file instead of to the screen;

    • Using pipes to send the output of one utility directly to another utility so that you can solve problems right on the command line;

    • Running programs in the background so that you can work on one task while UNIX is working on a different one; and

    • Using the shell to generate filenames to save you time spent on typing and help you when you do not recall the exact name of a file.

Part II: The Editors

Part II covers two classic, powerful UNIX command line text editors. Mac OS X includes the vim text editor, an "improved" version of the widely used vi editor, as well as the popular GNU emacs editor. Text editors enable you to create and modify text files that can hold programs, shell scripts, memos, and input to text formatting programs.

  • Chapter 6 The vim Editor

    Starts with a tutorial on vim and then explains how to use many of the advanced features of vim, including special characters in search strings, the General-Purpose and Named buffers, parameters, markers, and execution of commands from vim. The chapter concludes with a summary of vim commands.

  • Chapter 7 The emacs Editor

    Opens with a tutorial and then explains many of the features of the emacs editor as well as how to use the META, ALT, and ESCAPE keys. The chapter also covers key bindings, buffers, and incremental and complete searching for both character strings and regular expressions. In addition, it details the relationship between Point, the cursor, Mark, and Region. It also explains how to take advantage of the extensive online help facilities available from emacs. Other topics covered include cutting and pasting, using multiple windows and frames, and working with emacs modesspecifically C mode, which aids programmers in writing and debugging C code. Chapter 7 concludes with a summary of emacs commands.

Part III: The Shells

Part III goes into more detail about bash and introduces the TC Shell (tcsh).

  • Chapter 8 The Bourne Again Shell

    Picks up where Chapter 5 leaves off, covering more advanced aspects of working with a shell. For examples it uses the Bourne Again Shellbash, the shell used almost exclusively for system shell scripts. Chapter 8 describes how to

    • Use shell startup (preference) files, shell options, and shell features to customize your shell;

    • Use job control to stop jobs and move jobs from the foreground to the background, and vice versa;

    • Modify and reexecute commands using the shell history list;

    • Create aliases to customize commands;

    • Work with user-created and keyword variables in shell scripts;

    • Set up functions, which are similar to shell scripts but can execute more quickly;

    • Write and execute simple shell scripts; and

    • Redirect error messages so that they go to a file instead of the screen.

  • Chapter 9 The TC Shell

    Describes tcsh and covers features that are common to and differ between bash and tcsh. This chapter explains how to

    • Run tcsh and change your default shell to tcsh;

    • Redirect error messages so that they go to files instead of to the screen;

    • Use control structures to alter the flow of control within shell scripts;

    • Work with tcsh array and numeric variables; and

    • Use shell builtin commands.

Part IV: More About Mac OS X

  • Chapter 10 Networking

    Discusses networks, network security, and the Internet. This chapter

    • Covers types of networks, subnets, protocols, addresses, hostnames, and various network utilities;

    • Discusses distributed computing including the client/server model; and

    • Describes briefly some of the servers you can use on a network.

  • Chapter 11 System Maintenance

    Discusses core system concepts including Superuser and the use of sudo as well as system operation. This chapter

    • Provides general information about how to set up a server, including information on property list and other configuration files;

    • Describes how the launchd superserver works;

    • Discusses the NetInfo database and delves into how it differs from the classic UNIX system databases;

    • Describes briefly many important standard directories and files;

    • Explains how to set up a chroot jail; and

    • Describes how DHCP works.

Part V: Programming Tools

Part V covers programming under Mac OS X. It discusses the C programming environment, the use of bash as a programming language, and ways to write programs using awk and sed.

  • Chapter 12 Programming Tools

    Introduces Mac OS X's exceptional programming environment. This chapter focuses on the traditional UNIX command line development environment because this environment is the one you will likely have to work in with free software developed for UNIX systems. This chapter

    • Discusses the Carbon, Cocoa, and UNIX APIs;

    • Explains how to invoke the GNU gcc compiler;

    • Describes how to use make to keep a set of programs up-to-date;

    • Explains how to debug a C program using gdb;

    • Describes how to work with shared libraries;

    • Explains how to set up and use CVS to manage and track program modules in a software development project; and

    • Discusses system calls and explains how you can use them to initiate kernel operations.

  • Chapter 13 Programming the Bourne Again Shell

    Once you have mastered the basics of OS X, you can use your knowledge to build more complex and specialized programs, by using the shell as a programming language. Chapter 13 shows you how to use bash to write advanced shell scripts. It discusses

    • Control structures such as if...then...else and case;

    • Variables, including locality of variables;

    • Arithmetic and logical (Boolean) expressions; and

    • Some of the most useful shell builtin commands, including exec, trap, and getopts.

    Chapter 13 poses two complete shell programming problems and then shows you how to solve them step by step. The first problem uses recursion to create a hierarchy of directories. The second problem develops a quiz program and shows you how to set up a shell script that interacts with a user and how the script processes data. (The examples in Part VI also demonstrate many features of the utilities you can use in shell scripts.)

  • Chapter 14 The awk Pattern Processing Language

    Explains how to write programs using the powerful awk language that filter data and write reports. The advanced programming section describes the powerful getline statement, which gives you better control over bringing data into awk and the system statement, which allows you to run external commands from inside an awk program.

  • Chapter 15 The sed Editor

    Describes sed, the noninteractive stream editor that finds many applications as a filter within shell scripts. This chapter discusses how to use sed's buffers to write simple yet powerful programs and includes many examples.

Part VI: Command Reference

Mac OS X includes hundreds of utilities. Chapters 14 and 15 as well as Part VI provide extensive examples of the use of more than 90 of the most important utilities with which you can solve problems without resorting to programming in C. If you are already familiar with UNIX, this part of the book will be a valuable, easy-to-use reference. If you are not an experienced user, it will serve as a useful supplement while you are mastering the earlier sections of the book.

Although the descriptions of the utilities in Chapters 14 and 15 and Part VI are presented in a format similar to that used by the Mac OS X manual (man) pages, they are much easier to read and understand. These utilities were chosen because you will work with them day in and day out (for example, ls and cp), because they are powerful tools that are especially useful in shell scripts (sort, paste, and test), because they help you work with your Mac OS X system (ps, kill, sysctl, and GetFileInfo), or because they enable you to communicate with other systems (ssh, scp, and ftp). Part VI also covers several utilities that were developed by Apple for OS Xthey are not found on other UNIX systems (nidump, otool, launchctl, and plutil).

Each utility description includes complete explanations of its most useful options. The "Discussion" and "Notes" sections present tips and tricks for using the utility to full advantage. The "Examples" sections demonstrate how to use these utilities in real life, alone and together with other utilities to generate reports, summarize data, and extract information. Take a look at the "Examples" sections for awk (more than 20 pages, starting on page 609), ftp (page 738), and sort (page 837) to see how extensive these sections are.

Part VII: Appendixes

Part VII includes the appendixes and the glossary.

  • Appendix A Regular Expressions

    Explains how to use regular expressions to take advantage of the hidden power of UNIX. Many utilities, including grep, sed, vim, and awk, accept regular expressions in place of simple strings of characters. A single regular expression can match many simple strings.

  • Appendix B Help

    Details the steps typically used to solve the problems you may encounter with a Mac OS X/UNIX system. This appendix also includes many links to Web sites that offer documentation, useful UNIX information, mailing lists, and software.

  • Appendix C Mac OS X for UNIX Users

    This appendix is a brief guide to Mac OS X features and quirks that may be unfamiliar to users who have been using other UNIX or UNIX-like systems.

  • Glossary

    Defines more than 500 terms that pertain to the use of Mac OS X/UNIX.

The index helps you quickly find the information you want.


Supplements

Mark Sobell's home page (www.sobell.com) contains downloadable listings of the longer programs from this book as well as pointers to many interesting and useful UNIX-related sites on the World Wide Web, a list of corrections to the book, answers to even-numbered exercises, and a solicitation for corrections, comments, and suggestions.


Thanks from Mark G. Sobell

First and foremost I want to thank my editor at Prentice Hall, Mark L. Taub, who encouraged me and kept me on track. His comments and direction were invaluable. Thank you, Mark T.

A big "Thank You" to the folks who read through the drafts of the book and made comments that caused me to refocus parts of the book where things were not clear or were left out altogether: Thank you to David Chisnall, computer scientist extraordinaire; Sean Fagan, Apple Computer; Paul M. Lambert, Apple Computer; Nicolas Roard; Stephen Worotynec, Engineering Services, Alias Systems; Gretchen Phillips, Independent Consultant, GP Enterprises; and Peggy Fenner.

Thanks to Molly Sharp of ContentWorks, the production manager who made sure the book came together as it was supposed to; Jill Hobbs, the copy editor who kept the various parts of the English language in their proper relative places; and Andrea Fox, the most amazing proofreader I have ever worked with. Thanks also to the folks at Prentice Hall who helped bring this book to life: Kim Silvestro, Publicist; Stephane Nakib, Marketing Manager; Julie Nahil, Full-Service Production Manager; Noreen Regina, Editorial Assistant; and everyone else who worked behind the scenes to make this book happen.

I am also indebted to Denis Howe, the editor of The Free On-Line Dictionary of Computing (FOLDOC). Dennis has graciously permitted me to use entries from his compilation. Be sure to visit the dictionary (www.foldoc.org).

Dr. Brian Kernighan and Rob Pike graciously allowed me to reprint the bundle script from their book, The UNIX Programming Environment (Prentice Hall, 1984).

Parts of A Practical Guide to UNIX® for Mac OS® X Users have grown from my UNIX and Linux books, and I want to thank the people who helped with those books.

Thank you to Lars Kellogg-Stedman, Harvard University; Jim A. Lola, Principal Systems Consultant, Privateer Systems, LLC; Eric S. Raymond, cofounder, Open Source Initiative; Scott Mann; Randall Lechlitner, Independent Computer Consultant; Jason Wertz, Computer Science Instructor, Montgomery County Community College; Justin Howell, Solano Community College; Ed Sawicki, The Accelerated Learning Center; David Mercer, Contechst; Jeffrey Bianchine, Advocate, Author, Journalist; John Kennedy; Chris Karr; Jim Dennis, Starshine Technical Services; Carsten Pfeiffer, Software Engineer and KDE Developer; Aaron Weber, Ximian; Matthew Miller, Boston University; Cristof Falk, Software Developer at Critter-Design; Scott Mann, IBM, Systems Management and Integration Professional; Steve Elgersma, Computer Science Department, Princeton University; Scott Dier, University of Minnesota; and Robert Haskins, Computer Net Works.

Thanks also to Dustin Puryear, Puryear Information Technology; Gabor Liptak, Independent Consultant; Bart Schaefer, Chief Technical Officer, iPost; Michael J. Jordan, Web Developer, Linux Online Inc.; Steven Gibson, Owner of SuperAnt.com; John Viega, Founder and Chief Scientist, Secure Software, Inc.; K. Rachael Treu, Internet Security Analyst, Global Crossing; Kara Pritchard, K & S Pritchard Enterprises, Inc.; Glen Wiley, Capitol One Finances; Karel Baloun, Senior Software Engineer, Looksmart, Ltd.; Matthew Whitworth; Dameon D. Welch-Abernathy, Nokia Systems; Josh Simon, Consultant; Stan Isaacs; and Dr. Eric H. Herrin II, Vice President, Herrin Software Development, Inc. And thanks to Doug Hughes, long-time system designer and administrator, who gave me a big hand with the sections on system administration, networks, the Internet, and programming.

More thanks go to consultants Lorraine Callahan and Steve Wampler; Ronald Hiller, Graburn Technology, Inc.; Charles A. Plater, Wayne State University; Bob Palowoda; Tom Bialaski, Sun Microsystems; Roger Hartmuller, TIS Labs at Network Associates; Kaowen Liu; Andy Spitzer; Rik Schneider; Jesse St. Laurent; Steve Bellenot; Ray W. Hiltbrand; Jennifer Witham; Gert-Jan Hagenaars; and Casper Dik.

A Practical Guide to UNIX® for Mac OS® X Users is based in part on two of my early UNIX books: UNIX System V: A Practical Guide and A Practical Guide to the UNIX System. Many people helped me with those books, and thanks here go to Pat Parseghian, Dr. Kathleen Hemenway, and Brian LaRose; Byron A. Jeff, Clark Atlanta University; Charles Stross; Jeff Gitlin, Lucent Technologies; Kurt Hockenbury; Maury Bach, Intel Israel Ltd.; Peter H. Salus; Rahul Dave, University of Pennsylvania; Sean Walton, Intelligent Algorithmic Solutions; Tim Segall, Computer Sciences Corporation; Behrouz Forouzan, DeAnza College; Mike Keenan, Virginia Polytechnic Institute and State University; Mike Johnson, Oregon State University; Jandelyn Plane, University of Maryland; Arnold Robbins and Sathis Menon, Georgia Institute of Technology; Cliff Shaffer, Virginia Polytechnic Institute and State University; and Steven Stepanek, California State University, Northridge, for reviewing the book.

I continue to be grateful to the many people who helped with the early editions of my UNIX books. Special thanks are due to Roger Sippl, Laura King, and Roy Harrington for introducing me to the UNIX system. My mother, Dr. Helen Sobell, provided invaluable comments on the original manuscript at several junctures. Also, thanks go to Isaac Rabinovitch, Professor Raphael Finkel, Professor Randolph Bentson, Bob Greenberg, Professor Udo Pooch, Judy Ross, Dr. Robert Veroff, Dr. Mike Denny, Joe DiMartino, Dr. John Mashey, Diane Schulz, Robert Jung, Charles Whitaker, Don Cragun, Brian Dougherty, Dr. Robert Fish, Guy Harris, Ping Liao, Gary Lindgren, Dr. Jarrett Rosenberg, Dr. Peter Smith, Bill Weber, Mike Bianchi, Scooter Morris, Clarke Echols, Oliver Grillmeyer, Dr. David Korn, Dr. Scott Weikart, and Dr. Richard Curtis.

We take responsibility for any errors and omissions in this book. If you find one or just have a comment, let us know (mgs@sobell.com) and we will fix it in the next printing. Mark Sobell's home page (www.sobell.com) contains a list of errors and credits those who found them. It also offers copies of the longer scripts from the book and pointers to many interesting Mac OS X Web sites.


Chapter 1. Welcome to Mac OS X

Previous Page
Next Page

1. Welcome to Mac OS X

IN THIS CHAPTER

The Heritage of Mac OS X: UNIX | 2
The Open Source Connection | 3
The Mach Kernel | 3
The GNU Project | 4
The BSD UNIX System | 5
Darwin | 5
Standards | 11

Mac OS X (The "X" is pronounced "ten") is the descendant of two operating systems: the classic Mac OS and the NeXTSTEP operating system. Mac OS X is much more closely related to NeXTSTEP, even though the application environment and user interface are more familiar to Macintosh users. This chapter begins by providing a brief history of Mac OS X; the rest of the chapter fills in some of the details.

The Mac OS debuted in 1984 as a single-tasking system. The first multitasking support appeared in Mac OS 7 (called System 7, not related to UNIX Version 7), released in 1990. By the time Mac OS 8 and 9 came out, the writing was on the wall: Mac OS needed to be revamped. In fact, discussions of the upcoming new operating system occupied Mac advocates and debaters for many happy years, starting shortly after Mac OS 7 was released. Some of the technology and application program interfaces (APIs, page 921) developed for the planned new system were supported under Mac OS 8 and 9. They became the Carbon interface, used to provide compatibility between classic Mac OS and Mac OS X.

Meanwhile, in the late 1980s, Steve Jobs began developing a computer that was similar to the Mac in many ways, named the NeXT. The NeXT was generally regarded as technically brilliant but too expensive and too slow. The first systems, which were released in 1988, used special removable optical media drives instead of conventional hard disks. The NeXT platform, called NeXTSTEP (with varying capitalization from one year to the next), was one of the first adopters of the Mach (pronounced "maak") microkernel developed at Carnegie Mellon University (CMU) in the mid-1980s. The NeXT used BSD UNIX tools and utilities, and it offered a BSD UNIX API for developers. The NeXT also had its own development environment, based on Objective-C, which was built around the GNU C compiler and GNU development tools The system also used some GNU utilities. NEXTSTEP (with the capitalization changed yet again) was the basis for a new API named OpenStep, which was shared with Sun. The OPENSTEP system (NeXT's implementation of the OpenStep API) became the new NeXT platform.

In 1996 Apple hired Steve Jobs as CEO, and in 1997 Apple bought NeXT. The various attempts to redesign Mac OS were over: The NeXT operating system was to become the new operating system for the Macintosh. Mac OS X was first released commercially in 1999, when Mac OS X Server hit the market. The first retail release of Mac OS X aimed at end users was in early 2001, but it was not until late 2001 that the system was stable enough for widespread adoption.

Mac OS X release 10.4 is still based on the Mach microkernel, the GNU development tools, the BSD UNIX utilities and API, and recognizable pieces of the NeXT user interface. The UNIX utilities have been updated from 4.3BSD to 4.4BSD, with pieces of NetBSD, FreeBSD, and 4.4BSD-Lite release 2 added. Today, Mac OS X mostly tracks FreeBSD's utilities and libraries.

Several major releases of Mac OS X have occurred, each with the code name of some large feline. In order, they are 10.0 (Cheetah), 10.1 (Puma), 10.2 (Jaguar), 10.3 (Panther), and 10.4 (Tiger). Two versions of Mac OS X are available: Mac OS X and Mac OS X Server.

The Darwin system, which is the foundation for Mac OS X, has version numbers that correspond to Mac OS X version numbers. Mac OS X 10.3 is based on Darwin 7.0, and Mac OS X 10.4 is based on Darwin 8.0. Minor version numbers follow a similar pattern: Darwin 8.2 code is used in Mac OS X 10.4.2.


The Heritage of Mac OS X: UNIX

UNIX was developed by researchers who needed a set of modern computing tools to help them with their projects. The system allowed a group of people working together on a project to share selected data and programs while keeping other information private.

Universities and colleges played a major role in furthering the popularity of UNIX through the "four-year effect." When the UNIX operating system became widely available in 1975, Bell Labs offered it to educational institutions at nominal cost. The schools, in turn, used it in their computer science programs, ensuring that computer science students became familiar with it. Because UNIX was such an advanced development system, the students became acclimated to a sophisticated programming environment. As these students graduated and went into industry, they expected to work in a similarly advanced environment. As more of these students worked their way up in the commercial world, the UNIX operating system found its way into industry.

BSD (Berkeley) UNIX

In addition to introducing students to UNIX, the Computer Systems Research Group (CSRG) at the University of California at Berkeley made significant additions and changes to the operating system. They made so many popular changes that one of the versions of the system is called the Berkeley Software Distribution (BSD) of the UNIX system (or just Berkeley UNIX). The other major version is UNIX System V, which descended from versions developed and maintained by AT&T and UNIX System Laboratories. Mac OS X inherits much more strongly from the BSD branch of the tree.


The Open Source Connection

An operating system is the low-level software that schedules tasks, allocates storage, and handles the interfaces to peripheral hardware, such as printers, disk drives, the screen, keyboard, and mouse. An operating system has two main parts: the kernel and the system programs. The kernel allocates machine resources, including memory, disk space, and CPU (page 928) cycles, to all other programs that run on the computer. The system programs perform higher-level housekeeping tasks, often acting as servers in a client/server relationship.

Much of the Mac OS X software is open source. Mac OS X uses the Mach kernel, many utilities come from the GNU project, and system programs come mostly from BSD code, although Apple has developed a number of new programs.

The Mach Kernel

The Mach kernel was developed as a research project at Carnegie Mellon University. CMU concluded its work on the project in 1994, although other groups have continued this line of research. The version of Mach used in Mac OS X has been under internal development at Apple for a long time, so its version numbers are not directly comparable with version numbers from the research projects.

Mach employs a microkernel architecture. While traditional UNIX systems favor a single giant kernel that contains all of the code for the operating system, Mach is a very small kernel that provides only minimal features. Other programs provide core operating system services. For example, on a traditional UNIX system the virtual memory system is part of the kernel. In contrast, under Mac OS X running the Mach kernel, parts of the virtual memory system are run by programs that are separate from the kernel.

The GNU Project

In 1983, Richard Stallman (www.stallman.org) announced[1] the GNU Project for creating an operating system, both kernel and system programs, and presented the GNU Manifesto,[2] which begins as follows:

[1] www.gnu.org/gnu/initial-announcement.html

[2] www.gnu.org/gnu/manifesto.html

GNU, which stands for Gnu's Not UNIX, is the name for the complete UNIX-compatible software system which I am writing so that I can give it away free to everyone who can use it.

Some years later, Stallman added a footnote to the preceding sentence when he realized that it was creating confusion:

The wording here was careless. The intention was that nobody would have to pay for *permission* to use the GNU system. But the words don't make this clear, and people often interpret them as saying that copies of GNU should always be distributed at little or no charge. That was never the intent; later on, the manifesto mentions the possibility of companies providing the service of distribution for a profit. Subsequently I have learned to distinguish carefully between "free" in the sense of freedom and "free" in the sense of price. Free software is software that users have the freedom to distribute and change. Some users may obtain copies at no charge, while others pay to obtain copiesand if the funds help support improving the software, so much the better. The important thing is that everyone who has a copy has the freedom to cooperate with others in using it.

The GNU Project has grown considerably since Stallman's announcement. Today GNU is run by the Free Software Foundation (FSF), which Stallman also founded.

GPL

GNU software is distributed under the terms of the GNU General Public License (GPL, www.gnu.org/licenses/licenses.html). The GPL says you have the right to copy, modify, and redistribute the code covered by the agreement. When you redistribute the code, however, you must also distribute the same license with the code, making the code and the license inseparable. If you get source code off the Internet for an accounting program that is under the GPL and modify that code and redistribute an executable version of the program, you must also distribute the modified source code and the GPL agreement with it. Because this is the reverse of the way a normal copyright works (it gives rights instead of limiting them), it has been termed a copyleft. (This paragraph is not a legal interpretation of the GPL; it is here merely to give you an idea of how it works. Refer to the GPL itself when you want to make use of it.)

Mac OS X uses the GNU C compiler, gcc, which includes a language named Objective-C. Objective-C, often called ObjC, is a reflective, object-oriented programming language. In addition, Mac OS X takes advantage of other GNU development tools.

The BSD UNIX System

In 1989, the University of California at Berkeley released parts of the BSD UNIX system under a very permissive license, requiring only that credit to the original authors be given. During the next few years, more code was rewritten without any trace of AT&T's original UNIX code in it until a reasonably complete operating system, 4.4BSD, was available for distribution. Bill Jolitz provided the few missing pieces and in 1989 began distributing 386/BSD, an operating system that ran on Intel 80386 processors. A commercial variant, BSD/OS, showed up almost immediately. In 1992, UNIX Systems Labs (the then-current owner of the UNIX license) sued, first BSDI (the vendor of BSD/OS), and then both BSDI and the University of California, over alleged copyright infringement; it took nearly two years to establish the fact that the code was clean and could be distributed.

Because Mac OS X uses many BSD utilities, it is generally considered a member of the BSD family. Unlike the other BSDs (such as NetBSD, FreeBSD, and OpenBSD), Mac OS X does not use a BSD kernel, instead using the Mach microkernel architecture.

Darwin

Darwin (opendarwin.org and developer.apple.com/darwin) is the open-source release of the underlying operating system code of Mac OS X, without the user interface and some proprietary drivers. Although Apple has only sold Macintosh computers based on Motorola and IBM processors, Darwin has been running on Intel processors for years. Apple has recently announced its intent to migrate Mac OS X to Intel processors. While this book talks about Mac OS X, much of the material applies equally to Darwin.


Overview of Mac OS X

The Mac OS X operating system has many unique and powerful features. Like other operating systems, Mac OS X is a control program for computers. But like UNIX, it is also a well-thought-out family of utility programs (Figure 1-1) and a set of tools that allow users to connect and use these utilities to build systems and applications.

Figure 1-1. A layered view of the Mac OS X operating system

Mac OS X Has a Kernel Programming Interface

The Mac OS X kernelthe heart of the Mac OS X operating systemis responsible for allocating the computer's resources and scheduling user jobs so that each one gets its fair share of system resources, including access to the CPU; peripheral devices, such as disk and CD-ROM storage; and printers. Programs interact with the kernel through system calls, special functions with well-known names. A programmer can use a single system call to interact with many kinds of devices. For example, there is one write( ) system call, not many device-specific ones. When a program issues a write( ) request, the kernel interprets the context and passes the request to the appropriate device. This flexibility allows old utilities to work with devices that did not exist when the utilities were originally written. It also makes it possible to move programs to new versions of the operating system without rewriting them (provided that the new version recognizes the same system calls).

Mac OS X Can Support Many Users

Depending on the hardware and the types of tasks that the computer performs, a Mac OS X system can support from 1 to more than 100 users, each concurrently running a different set of programs. The per-user cost of a computer that can be used by many people at the same time is less than that of a computer that can be used by only a single person at a time. It is less because one person cannot generally use all the resources a computer has to offer. That is, no one can keep all the printers going constantly, keep all the system memory in use, keep all the disks busy reading and writing, keep the Internet connection in use, and keep all the terminals busy at the same time. A multiuser operating system allows many people to use all the system resources almost simultaneously. The use of costly resources can be maximized and the cost per user can be minimizedthe primary objectives of a multiuser operating system.

Because Mac OS X is mostly run on personal desktop machines, many users do not take much advantage of these features, often getting nothing from Mac OS X's multiuser support except a graphically interesting user-switching feature and a little extra security. Concurrent use of a desktop system is rare, but network access to the command line makes it possible for multiple users to share a system, even though only one person at a time uses the console.

Mac OS X Can Run Many Tasks

Mac OS X is a fully protected multitasking operating system, allowing each user to run more than one job at a time. Processes can communicate with one another but remain fully protected from one another, just as the kernel remains protected from all processes. You can run several jobs in the background while giving all your attention to the job being displayed on the screen, and you can switch back and forth between jobs. And, of course, you can run multiple graphical applications. This capability ensures that users can be more productive.

Mac OS X Provides a Secure Hierarchical Filesystem

A file is a collection of information, such as text for a memo or report, an accumulation of sales figures, an image, a song, or an executable program. Each file is stored under a unique identifier on a storage device, such as a hard disk. The Mac OS X filesystem provides a structure whereby files are arranged under directories, which are like folders or boxes. (Apple documentation uses the terms folder and directory interchangeably.) Each directory has a name and can hold other files and directories. Directories, in turn, are arranged under other directories, and so forth, in a treelike organization. This structure helps users keep track of large numbers of files by grouping related files into directories. Each user has one primary directory and as many subdirectories as required (Figure 1-2).

Figure 1-2. The Mac OS X hierarchical filesystem

Links

A link allows a given file to be accessed by means of two or more names. The alternative names can be located in the same directory as the original file or in another directory. Links can make the same file appear in several users' directories, enabling them to share the file easily. Windows uses the term shortcut in place of link; Macintosh users will be more familiar with the term alias. The terms shortcut, link, and alias are generally interchangeable in this context. (UNIX users are used to the term alias referring to a command macro feature provided by the shells. This book uses the term alias to refer to command macros, not for links.)

Security

Like most multiuser operating systems, Mac OS X allows users to protect their data from access by other users. It also allows users to share selected data and programs with certain other users by means of a simple but effective protection scheme. This level of security is provided by file access permissions, which limit which users can read from, write to, or execute a file. Access Control Lists (ACLs) are available as of Mac OS X 10.4. ACLs give users and administrators finer-grained control over file access permissions.

The Shell: Command Interpreter and Programming Language

In a textual environment, the shellthe command interpreteracts as an interface between you and the operating system. When you enter a command on the screen, the shell interprets the command and calls the program you want. A number of shells are available for Mac OS X, including these two common ones:

  • The Bourne Again Shell (bash), an enhanced version of the Bourne Shell, one of the original UNIX shells

  • The TC Shell (tcsh), an enhanced version of the C Shell, developed as part of BSD UNIX

Because users often prefer different shells, multiuser systems can have a number of different shells in use at any given time. The choice of shells demonstrates one of the powers of the Mac OS X operating system: the ability to provide a customized user interface.

Besides performing its function of interpreting commands from a keyboard and sending those commands to the operating system, the shell is a high-level programming language. Shell commands can be arranged in a file for later execution (OS X calls these files shell scripts; DOS and Windows call them batch files). This flexibility allows users to perform complex operations with relative ease, often with rather short commands, or to build with surprisingly little effort elaborate programs that perform highly complex operations.

Filename Generation

When you are typing commands to be processed by the shell, you can construct patterns using characters that have special meanings to the shell. The characters are called wildcard characters. These patterns represent a kind of shorthand: Rather than typing in complete filenames, users can type in patterns and the shell will expand them into matching filenames. These patterns are called ambiguous file references. An ambiguous file reference can save you the effort of typing in a long filename or a long series of similar filenames. For example, the shell might expand the pattern mak* to make-3.80.tar.gz. Patterns can also be useful when you know only part of a filename or cannot remember the exact spelling. For example, the command ls *memo* lists all the filenames containing the string memo.

Device-Independent Input and Output

Redirection

Devices (such as a printer or terminal) and disk files appear as files to Mac OS X programs. When you give a command to Mac OS X, you can instruct it to send the output to any one of several devices or files. This diversion is called output redirection.

In a similar manner, a program's input that normally comes from a keyboard can be redirected so that it comes from a disk file instead. Input and output are device independent; they can be redirected to or from any appropriate device.

As an example, the cat utility normally displays the contents of a file on the screen. When you run a cat command, you can easily cause its output to go to a disk file instead of the screen.

Shell Functions

One of the most important features of the shell is that users can use it as a programming language. Because the shell is an interpreter, it does not compile programs written for it but rather interprets them each time they are loaded from the disk. Loading and interpreting programs can be time-consuming.

Many shells, including the Bourne Again Shell, include shell functions that the shell holds in memory so that it does not have to read them from the disk each time you want to execute them. The shell also keeps functions in an internal format so that it does not have to spend as much time interpreting them.

Job Control

Job control is a shell feature that allows users to work on several jobs at once, switching back and forth between them as desired. When you start a job, it is frequently in the foreground so that it is connected to your terminal. Using job control, you can move the job you are working with into the background and continue running it there while working on or observing another job in the foreground. If a background job then needs your attention, you can move it into the foreground so that it is once again attached to the terminal. The concept of job control originated with BSD UNIX, where it appeared in the C Shell.

A Large Collection of Useful Utilities

Mac OS X includes a family of several hundred utility programs, often referred to as commands. These utilities perform functions that are universally required by users. An example is sort. The sort utility puts lists (or groups of lists) in alphabetical or numerical order and can be used to sort by part number, last name, city, ZIP code, telephone number, age, size, cost, and so forth. The sort utility is an important programming tool and is part of the standard Mac OS X system. Other utilities allow users to create, display, print, copy, search, and delete files as well as to edit, format, and typeset text. Most of the utilities are installed by default under Mac OS X, but some (such as those in the Developer Tools package) must be installed separately. The man (for manual) and info utilities provide online documentation of Mac OS X itself, as does the GUI Help application available from the ubiquitous Help menu.

Interprocess Communication

Pipes and filters

Mac OS X allows users to establish both pipes and filters on the command line. A pipe sends the output of one program to another program as input. A filter is a special form of a pipe that processes a stream of input data to yield a stream of output data. A filter processes another program's output, altering it as a result. The filter's output then becomes input to another program.

Pipes and filters frequently join utilities to perform a specific task. For example, you can use a pipe to send the output of the cat utility to sort, a filter, and then use another pipe to send the output of sort to a third utility, lpr, that sends the data to a printer. Thus, in one command line, you can use three utilities together to sort and print a file.

System Maintenance and Administration

On an OS X system, the system maintainer or administrator is frequently the owner and only user of the system. This person has many responsibilities. The first responsibility may be to set up the system and install the software.

Once the system is up and running, the system maintainer is responsible for downloading and installing software (including upgrading the operating system), backing up and restoring files, and managing such system facilities as printers, terminals, servers, and a local network. The system maintainer is also responsible for setting up accounts for new users on a multiuser system, bringing the system up and down as needed, and taking care of any problems that arise.


Additional Features of Mac OS X

The Mac OS X developers included features from BSD, System V, Sun Microsystems' Solaris, and Linux, as well as new features, in their operating system. Although most of the tools found on UNIX exist for Mac OS X, in some cases these tools have been replaced by more modern counterparts. This section describes many of the popular tools and features available under OS X.

GUIs: Graphical User Interfaces

Aqua

Mac OS X comes with two graphical interfaces that can be used simultaneously. Macintosh users are familiar with Aqua, the standard Mac OS X graphical interface. Aqua is based on a rendering technology called Quartz and has a standard look and feel for applications.

X11

Mac OS X also supports the X Window System (also called X or X11). The X in X Window System has nothing to do with the X in Mac OS X. X11 was developed in part by researchers at MIT (Massachusetts Institute of Technology) and provides the foundation for the GUIs available with many UNIX systems. Because X11 is widely used on other systems, support for X11 applications under Mac OS X allows users greater flexibility and access to a large pool of applications developed for UNIX (and Linux) systems. Given a terminal or workstation screen that supports X11, a user can interact with the computer through multiple windows on the screen, display graphical information, or use special-purpose applications to draw pictures, monitor processes, or preview formatted output. X11 is an across-the-network protocol that allows a user to open a window on a workstation or computer system that is remote from the CPU generating the window.

Desktop manager

Usually two layers run under X11: a desktop manager and a window manager. A desktop manager is a picture-oriented user interface that enables you to interact with system programs by manipulating icons instead of typing the corresponding commands to a shell. Under Mac OS X, the Finder provides the desktop while Aqua launches applications and handles system menus, so there is no need for a desktop manager.

Mac OS X handles the desktop in Aqua, not in X11, so there is no desktop manager under X11.

Window manager

A window manager is a program that allows you to open and close windows, run programs, and set up a mouse so it does various things depending on how and where you click. The window manager also gives the screen its personality. Whereas Microsoft Windows allows you to change the color of key elements in a window, a window manager under X11 allows you to customize the overall look and feel of the screen: change the way a window looks and works (by giving it different borders, buttons, and scrollbars), set up virtual desktops, create menus, and more. Under Mac OS X, the default window manager is quartz-wm. This window manager, which manages X11 applications, mimics the look and feel of the Apple Aqua user interface so as to better integrate X11 applications with the Aqua desktop.

(Inter)Networking Utilities

Mac OS X network support includes many utilities that enable you to access remote systems over a variety of networks. In addition to sending email to users on other systems, you can access files on disks mounted on other computers as if they were located on the local system, make your files available to other systems in a similar manner, copy files back and forth, run programs on remote systems while displaying the results on the local system, and perform many other operations across local area networks (LANs) and wide area networks (WANs), including the Internet.

Layered on top of this network access is a wide range of application programs that extend the computer's resources around the globe. You can carry on conversations with people throughout the world, gather information on a wide variety of subjects, and download new software over the Internet quickly and reliably.

Standards

POSIX

Individuals from companies throughout the computer industry have joined together to develop the POSIX (Portable Operating System Interface for Computer Environments) standard, which is based largely on the UNIX System V Interface Definition (SVID) and other earlier standardization efforts. These efforts have been spurred by the U.S. federal government, which needs a standard computing environment to minimize its training and procurement costs. Now that these standards are gaining acceptance, software developers are able to develop applications that run on all conforming versions of UNIX, Linux, and other operating systems. The OS X BSD API allows developers to take advantage of these standards.

The C and Objective-C Programming Languages

Ken Thompson wrote the UNIX operating system in 1969 in PDP-7 assembly language. Assembly language is machine dependent: Programs written in assembly language work on only one machine or, at best, one family of machines. The original UNIX operating system therefore could not easily be transported to run on other machines (it was not portable).

To make UNIX portable, Thompson developed the B programming language, a machine-independent language, from the BCPL language. Dennis Ritchie developed the C programming language by modifying B and, with Thompson, rewrote UNIX in C in 1973. After this rewrite, the operating system could be transported more easily to run on other machines.

That was the start of C. Its roots reveal some of the reasons why it is such a powerful tool. C can be used to write machine-independent programs. A programmer who designs a program to be portable can easily move it to any computer that has a C compiler. C is also designed to compile into very efficient code. With the advent of C, a programmer no longer had to resort to assembly language to get code that would run efficiently.

C is a good systems language. You can write a compiler or an operating system in C. It is highly structured, but it is not necessarily a high-level language. C allows a programmer to manipulate bits and bytes, as is necessary when writing an operating system, but it also has high-level constructs that allow for efficient, modular programming.

In the late 1980s, the American National Standards Institute (ANSI) defined a standard version of the C language, commonly referred to as ANSI C or C89 (for the year the standard was published). Ten years later, the C99 standard was published; C99 is mostly supported by the GNU Project's C compiler (named gcc). The original version of the language is often referred to as Kernighan & Ritchie (or K & R) C, named for the authors of the book that first described the C language.[3]

[3] See cm.bell-labs.com/cm/cs/who/dmr/chist.html for a history of the C programming language.

As object-oriented programming became more popular, variants of C that supported object-oriented programming were developed. Two of the most famous are C++, developed at Bell Labs, and Objective-C, developed at a software company named Stepsone.

A researcher at Bell Labs, Bjarne Stroustrup, created the object-oriented programming language named C++, which is built on the foundation of C. Because object-oriented programming is desired by many employers today, C++ is preferred over C in many environments. Some developers felt that C++ did not do a good job of melding C and object-oriented concepts.

The Objective-C language is more accurately termed object-oriented. Objective-C was the language chosen for development on the NeXT. The NeXT compiler code was eventually contributed back to the GNU Project, and today the GNU C compiler supports Objective-C. Much of Mac OS X's application framework is written in Objective-C.

Apple continues to maintain versions of the GNU Project's C compiler and tools as the native ones used to build the Mac OS X operating system. The compiler and tools are distributed with Mac OS X.

Software Development

One of the strengths of Mac OS X is its rich software development environment. You can find compilers and interpreters for many computer languages. Besides C, Objective-C, and C++, other languages available for Mac OS X include Java, Perl, and Python. The tools that come with the system are just part of the story; many others are available for download or purchase. The bison utility generates parsing code that makes it easier to write programs to build compilers (tools that parse files containing structured information). The flex utility generates scanners (code that recognizes lexical patterns in text). The make utility and GNU's automatic configuration utility (configure) make it easy to manage complex development projects. Source code management systems, such as CVS, simplify version control. The gdb debugger helps in tracking down and repairing software defects. The GNU C compiler (gcc) works with the gprof profiling utility to help programmers identify potential bottlenecks in a program's performance. The C compiler includes options to perform extensive checking of C code that can make the code more portable and reduce debugging time. Finally Apple's Xcode development environment provides a unified graphical front end to most of these tools as well as other options and features.


Chapter Summary

The Mac OS X operating system grew out of the NeXT environment's combination of UNIX heritage and the Mach microkernel. Today it is one of the most widely used UNIX-like operating systems in the world. UNIX users will find a familiar environment in Mac OS X. Mac OS X contains the expected complement of UNIX utilities, contributed by programmers around the world, including many of the tools developed as part of the GNU Project. Apple is committed to the continued development of the system. With many commercial software packages available to run on Mac OS X, it is clear that the system has evolved into a serious competitor, both to other desktop environments, such as classic Mac OS and Windows, and to server and workstation UNIX systems.


Exercises

1. | What are multiuser systems? Why are they successful?
2. | What is the Free Software Foundation/GNU? BSD? Mach? Which parts of the Mac OS X operating system did each provide?
3. | What is a shell? How does it work with the kernel? With the user?
4. | How can you use utility programs and a shell to create your own applications?
5. | Why is the Mac OS X filesystem referred to as hierarchical?
6. | What is the difference between a multiprocessor and a multiprocessing system?
7. | Give an example of when you would want to use a multiprocessing system.
8. | What is a utility program?
9. | What are the key terms of the GNU General Public License?


Part I: The Mac OS X Operating System

CHAPTER 2 GETTING STARTED

CHAPTER 3 THE MAC OS X UTILITIES

CHAPTER 4 THE MAC OS X FILESYSTEM

CHAPTER 5 THE SHELL


Chapter 2. Getting Started

Previous Page
Next Page

2. Getting Started

IN THIS CHAPTER

Conventions Used in This Book | 18
Graphical Login | 20
Textual Login | 21
Curbing Your Power: Superuser/root/Administrator Access | 23
Working with the Shell | 23
Getting the Facts: Where to Find Documentation | 27
man: Displaying the System Manual | 28
info: Displaying Information About Utilities | 30
More About Logging In | 34
What to Do If You Cannot Log In | 34

One way or another you are sitting in front of a screen that is connected to a computer that is running Mac OS X. You may be working with a graphical user interface (GUI) or a textual interface. This book is mostly about the textual, or command line, interface to Mac OS X. If you are working with a GUI, you will need to use a terminal emulator application such as Terminal to follow along with the examples in this book.

This chapter starts with a discussion of the typographical conventions used in this book, followed by a section on logging in on the system. Next there is a brief reminder about the powers of Superuser (root) and how to avoid making mistakes that will make your system inoperable or hard to work with. The chapter continues with a discussion of where to find more information about Mac OS X. It concludes with additional information on logging in, including how to change your password.

While heeding the warning about the dangers of misusing the powers of Superuser on page 23, feel free to experiment with your system: Give commands, create files, follow the examples in this book, and have fun.


Conventions Used in This Book

This book uses conventions to make its explanations shorter and clearer. The following paragraphs describe these conventions.

Mac OS X

The term Mac OS X refers to both Mac OS X and Mac OS X Server. Important differences between the two are noted.

Mac OS X versions

This book is based on Mac OS X 10.4. Significant differences between versions 10.3 and 10.4 are noted. Because the book focuses on the underlying UNIX operating system, which changes little from one release of OS X to the next, the text will remain relevant through several future releases. The author's Web site (www.sobell.com) provides corrections and updates as appropriate.

Text and examples

The text is set in this type, whereas examples are shown in a monospaced font (also called a fixed-width font):

$ cat practice
This is a small file I created
with a text editor.

The next paragraph explains why part of the first line is in a bold typeface.

Items you enter

Everything you enter at the keyboard is shown in a bold typeface: Within the text, this bold typeface is used; within examples and screens, this one is used. In the previous example, the dollar sign ($) on the first line is a prompt that OS X displays, so it is not bold; the remainder of the first line is entered by a user, so it is bold.

Utility names

Names of utilities are printed in this bold sans serif typeface. This book references the emacs editor and the ls utility or ls command (or just ls) but instructs you to enter ls a on the command line. In this way the text distinguishes between utilities, which are programs, and the instructions you give on the command line to invoke the utilities.

Filenames

Filenames appear in a bold typeface. Examples are memo5, letter.1283, and reports. Filenames may include uppercase and lowercase letters; however, Mac OS X can be case sensitive (page 925), so memo5, MEMO5, and Memo5 may name three different files. By default these three filenames refer to the same file. For more information refer to "Filenames" on page 74.

Character strings

Within the text, characters and character strings are marked by putting them in a bold typeface. This convention avoids the need for quotation marks or other delimiters before and after a string. An example is the following string, which is displayed by the passwd utility: Mismatch; try again, EOF to quit.

Buttons and labels

Words appear in a bold typeface in the sections of the book that describe a GUI. This font indicates that you can click a mouse button when the mouse pointer is over these words on the screen or over a button with this name.

Keys and characters

This book uses SMALL CAPS for three kinds of items:

  • Important keyboard keys, such as the SPACE bar and the RETURN,[1] ESCAPE, and TAB keys.

    [1] Different keyboards use different keys to move the cursor (page 928) to the beginning of the next line. This book always refers to the key that ends a line as the RETURN key. Your keyboard may have a RET, NEWLINE, ENTER, RETURN, or other key. Some keyboards have a key with a bent arrow on it. (The key with the bent arrow is not an arrow key. Arrow keys have straight shafts.) Use the corresponding key on your keyboard each time this book asks you to press RETURN.

  • The characters that keys generate, such as the SPACEs generated by the SPACE bar.

  • Keyboard keys that you press with the CONTROL key, such as CONTROL-D. (Even though D is shown as an uppercase letter, you do not have to press the SHIFT key; enter CONTROL-D by holding the CONTROL key down and pressing d.)

Prompts and RETURNs

Most examples include the shell promptthe signal that Mac OS X is waiting for a commandas a dollar sign ($) or sometimes a percent sign (%). The prompt is not in boldface because you do not enter it. Do not type the prompt on the keyboard when you are experimenting with examples from this book. If you do, the examples will not work.

Examples omit the RETURN keystroke that you must use to execute them. An example of a command line is

$ vim memo.1204

To use this example as a model for running the vim editor, give the command vim memo.1204 and press the RETURN key. (Press ESCAPE ZZ to exit from vim; see page 147 for a vim tutorial.) This method of entering commands makes the examples in the book correspond to what appears on your screen.

Definitions

All glossary entries marked with FOLDOC are courtesy of Denis Howe, editor of the Free Online Dictionary of Computing (www.foldoc.org), and are used with permission. This site is an ongoing work containing definitions, anecdotes, and trivia.

Optional

Passages marked as optional appear in a gray box and are not central to the ideas presented in the chapter but often involve more challenging concepts. A good strategy when reading a chapter is to skip the optional sections and then return to them when you are comfortable with the main ideas presented in the chapter. This is an optional paragraph.

URLs (Web addresses)

Web addresses, or URLs, have an implicit http:// prefix, unless ftp:// or https:// is shown. You do not normally need to specify a prefix when the prefix is http:// but you must use a prefix from a browser when you specify an FTP or secure HTTP site. Thus you can specify a URL in a browser exactly as shown in this book.

Tip, caution, and security boxes

The following boxes highlight information that may be helpful while you are using or maintaining a Mac OS X system.

Tip: This is a tip box

A tip box may help you avoid repeating a common mistake or may point toward additional information.

Caution: This box warns you about something

A caution box warns you about a potential pitfall.

Security: This box marks a security note

A security box highlights a potential security issue. These notes are usually for system maintainers, but some apply to all users.


Logging In

Textual and graphical interfaces

There are two ways to work with a Mac OS X system: graphical and textual. When you are working on the system monitor, the monitor connected to or embedded in the system running OS X, you work with a graphical user interface by default. When you connect to the system from another system, a workstation, or a terminal, over a dial-up line or a network connection, you typically work with a textual interface, also called the command line interface (CLI).

You can use the terminal emulator application named Terminal (page 24) to display a textual window within a graphical environment. It is also possible to display a graphical interface over a network connection. This section has more information on the possibilities. The text console, described on page 35, displays a purely textual environment on the system monitor.

Graphical Login

When the system boots, the default behavior is to log in automatically the first user account that was created when the system was set up. No password is required. Once you log out, the system presents a list of accounts that you can log in to (Figure 2-1). When you select one of these accounts, the system prompts for a password and, when you supply the correct password and press RETURN, logs you in. You can deactivate the automatic login using the System Preferences application (page 27). You can also use System Preferences to cause OS X to display name and password text boxes (Figure 2-2) instead of the list of users. With this method of logging in, you must enter a username and password, pressing RETURN after each.

Figure 2-1. List of users method of logging in

Figure 2-2. Name and password text boxes method of logging in

Security: Always use a password

Unless you are the only user of your system; your system is not connected to any other systems, the Internet, or a modem; and you are the only one with physical access to your system, it is a bad idea to allow any user to log in without a password. One of the first steps in securing a system is to disable the automatic login feature (use System Preferences [page 27]).

Textual Login

Typically, when you log in remotely using a terminal, terminal emulator, or other text-based device on a Mac OS X system, you must enter your username and password in response to the system prompts, pressing RETURN after each (Figure 2-3).

Figure 2-3. Textual login
Darwin (bravo.example.com) (console)

login: zach
Password:
Last login: Mon Jul 18 16:59:14 from bravo.example.co
Welcome to Darwin!
You are using bash.
bravo:~ zach$

After you enter your username and password, the shell prompt (or just prompt) appears, indicating that you have successfully logged in; it indicates that the system is ready for you to give it a command. The shell prompt line may be preceded by a short message called the message of the day (kept in /etc/motd) and your last login information. By default OS X establishes a prompt of host:directory user$, where user is your username, host is the name of the local system, and directory is the name of the directory you are working in. For information on how to change your prompt, refer to page 286.

Security: Did you log in last?

When you perform a textual login, after you enter your username and password and press RETURN, the system displays information about the last login on your account, showing when it took place and where it originated. You can use this information to see whether anyone else has accessed this account since you last used it. If so, perhaps an unauthorized user has learned your password and has logged on as you. In the interest of security, advise the system administrator of the circumstances that made you suspicious and change your password (page 36).

Logging In Remotely: Terminal Emulation and ssh

When you are not using a console, terminal, or other device connected directly to the Mac OS X system you are logging in on, you are probably connected to the OS X system using terminal emulation software on another system. Running on the local computer, this software connects to the Mac OS X system via a network (Ethernet, asynchronous phone line, PPP, or other type) and allows you to log in on the Mac OS X system.

Caution: Make sure TERM is set correctly

No matter how you connect, make sure you have the TERM variable set to the type of terminal your emulator is emulating. For more help see "Specifying a Terminal" on page 913.

By default Mac OS X does not allow remote logins. You can enable remote logins via ssh by enabling remote login in the Services tab of the Sharing pane of the Preferences window (page 27). OS X does not support telnet logins.

When you log in via a dial-up line, the connection is straightforward: You instruct the local emulator program to contact the remote system, it dials the phone, and the remote OS X system displays a login prompt on the screen.

When you log in via a directly connected network, you use ssh (page 847) to connect to the remote system. You can use ssh from the command line or from a graphical environment. From the command line of a terminal or a terminal emulator of an Apple, PC, or UNIX-like machine, give the command ssh, followed by the name or IP address (refer to "Host Address" on page 393) of the system you want to log in on. For examples and more detail refer to "Running Commands from the Terminal Emulator/Shell" on page 24. For more information about logging in see "Textual Login" on page 21.


Curbing Your Power: Superuser/root/Administrator Access

While you are logged in as the user named root, you are referred to as Superuser or administrator and have extraordinary privileges. You can read from or write to any file on the system, execute programs that ordinary users cannot, and more. By default the root account is disabled (it has no password). Standard practice is to use sudo (page 431) to perform tasks that require root privileges.

On a multiuser system on which the root account has been enabled, you may not be permitted to know the root password, but someoneusually the system administratorknows the root password and maintains the system. When you are running OS X on your own computer, you can enable the root account (page 430). Refer to "System Administrator and Superuser" on page 427 for more information.

Caution: Do not experiment as Superuser

Feel free to experiment when you are logged in as yourself. When you log in as Superuser, also called root or administrator, or anytime you give the Superuser/root/administrator password or run sudo, do only what you have to do and make sure you know exactly what you are doing. When you have completed the task at hand, revert to working as yourself. When working as Superuser/root/administrator, you can damage the Mac OS X system to such a great extent that you will need to reinstall Mac OS X to get it working again. See "System Maintenance" on page 425 for tasks you may want to perform working as administrator.


Working with the Shell

The Mac OS X system displays a graphical interface by default. You need a shell prompt to enter commands from the CLI. To display a shell prompt, you can either run a program such as Terminal or you can connect to the system over a network.

Running the Mac OS X Command Line Interface

By default Mac OS X displays a graphical user interface. The UNIX operating system upon which OS X is based presents a textual, or command line, interface.

When you log in to a textual environment, such as from a terminal emulator or from the text console (page 35), the shell displays a prompt immediately. You are using the CLI. The easiest way to display a shell prompt from within a graphical environment is to use the Terminal utility (Figure 2-4), which is found in the Utilities folder in Applications.

Figure 2-4. A Terminal window

Tip: The command line interface

Although this section describes Terminal, it applies equally to the command line interface that OS X displays from a remote login or from the text console.

Running Commands from the Terminal Emulator/Shell

The Terminal utility (Figure 2-4) logs you in and runs a shell, which displays login messages and a prompt and waits for you to enter commands. This program is referred to as a terminal emulator, because it emulates the behavior of a serial terminal. By default Terminal emulates the behavior of a DEC vt100 terminal.

When you are typing in a terminal emulator window or using any CLI, several characters, including *, $, [, and ], have special meanings. Avoid typing these characters until you have read "Special Characters" on page 40.

The shell

When you are using a CLI, you are communicating with the command interpreter known as the shell. The shell plays an important part in much of your communication with Mac OS X. When you enter a command at the keyboard in response to the shell prompt on the screen, the shell interprets the command and initiates the appropriate action. This action may be executing your program, calling a compiler or a Mac OS X utility or another standard program, or displaying an error message telling you that you entered a command incorrectly. When you are working on a GUI, you bypass the shell and execute a program by clicking an icon or name. Refer to Chapter 5 (page 113) for more information on the shell.

Correcting Mistakes

This section explains how to correct typos and other errors you may make while you are logged in on a character-based display. Because the shell and most other utilities do not interpret the command line or other text until after you press RETURN, you can correct typing mistakes before you press RETURN.

You can correct typing mistakes in several ways: erase one character at a time, back up a word at a time, or back up to the beginning of the command line in one step. After you press RETURN, it is too late to correct a mistake; you can either wait for the command to run to completion or abort execution of the program.

Erasing a Character

While entering characters from the keyboard, you can back up and erase a mistake by pressing the erase key once for each character you want to delete. The erase key backs over as many characters as you wish. It does not, in general, back up past the beginning of the line.

Tip: CONTROL-Z suspends a program

Although not a way of correcting a mistake, you may press the suspend key (typically CONTROL-Z) by mistake and wonder what happened (you will see a message containing the word Stopped). You have just used job control (page 272) to stop your job. Give the command fg to continue your job in the foreground, and you should return to where you were before you pressed the suspend key. For more information refer to "bg: Sends a Job to the Background" on page 274.

The default erase key is DELETE. If this key does not work, try BACKSPACE or CONTROL-H. If these keys do not work, give the following stty[2] command to set the erase and line kill (see "Deleting a Line" below) keys to their defaults

[2] The command stty is an abbreviation for set teletypewriter, the first terminal that UNIX was run on. Today stty is commonly thought of as set terminal.

$ stty ek

Deleting a Word

You can delete a word you entered by pressing CONTROL-W. A word is any sequence of characters that does not contain a SPACE or TAB. When you press CONTROL-W, the cursor moves left to the beginning of the current word (as you are entering a word) or the previous word (when you have just entered a SPACE or TAB), removing the word.

Deleting a Line

Any time before you press RETURN, you can delete a line you are entering by pressing the line kill key, or kill key. When you press this key, the cursor moves to the left, erasing characters as it goes, back to the beginning of the line. The default line kill key is CONTROL-U. If this key does not work, try CONTROL-X. If these keys do not work, give the following command to set the erase and line kill keys to their defaults:

$ stty ek

Aborting Execution

Sometimes you may want to terminate a running program. A program may be performing a lengthy task such as displaying the contents of a file that is several hundred pages or copying a file that is not the one you meant to copy.

To terminate a program from a character-based display, press the interrupt key (CONTROL-C or sometimes DELETE or DEL). When you press this key, the operating system sends a terminal interrupt signal to the program you are running and to the shell. Exactly what effect this signal has depends on the program. Some programs stop execution immediately, whereas others ignore the signal. Some programs take other actions. When it receives a terminal interrupt signal, the shell displays a prompt and waits for another command.

If these methods do not terminate the program, try stopping the program with the suspend key (typically CONTROL-Z), giving the jobs command to verify the job number of the program, and using kill to abort the program. The job number is the number within the brackets at the left end of the line that jobs displays ([1]). The kill command uses TERM to send a termination signal[3] to the job specified by the job number, which is preceded by a percent sign (%1):

[3] When the terminal interrupt signal does not work, use the kill (KILL) signal. A running program cannot ignore kill; it is sure to abort the program (page 432).

$ bigjob
^Z
[1]+  Stopped                 bigjob
$ jobs
[1]+  Stopped                 bigjob
$ kill -TERM %1

[1]+  Stopped                 bigjob
$ RETURN
[1]+  Terminated              bigjob

The kill command returns a prompt; press RETURN again to see the confirmation message. For more information on job control refer to "Running a Program in the Background" on page 131. The kill command is very similar to the Force Quit option found on the Apple menu.

Repeating/Editing Command Lines

To repeat a previous command, press the UP ARROW key. Each time you press it, you see an earlier command line. To reexecute the displayed command line, press RETURN. Press DOWN ARROW to browse through the command lines in the other direction.

The RIGHT and LEFT ARROW keys move the cursor back and forth along the displayed command line. At any point along the command line, you can add characters by typing them. Use the erase key to remove characters from the command line. For more complex techniques in command line editing, see page 296.


Useful Graphical Tools

While this book does not spend much time discussing the OS X graphical environment, a few applications are worth mentioning as they can be of use while exploring the UNIX side of Mac OS X.

System Preferences

The System Preferences window (Figure 2-5) provides access to a number of system maintenance and configuration features. For example, System Preferences provides an easy way to change your password (page 36) and to handle system maintenance tasks (page 427) such as creating new accounts and enabling remote login via ssh.

Figure 2-5. The System Preferences window

[View full size image]

Panes

When you click a control icon in the System Preferences window, OS X replaces the System Preferences window with an interface for that control, called a pane. Click Show All, visible in the top button bar of all panes, to display the System Preferences window again.

On Mac OS X Server, many system maintenance functions are available from the Server Admin and Workgroup Manager applications.


Getting the Facts: Where to Find Documentation

Distributions of Mac OS X typically come with only a little hardcopy reference. However, online documentation has always been one of UNIX's strengths. The manual and info pages are available via the man and info utilities. With the growth of Mac OS X and the Internet, the sources of documentation have expanded. The following sections discuss some of the places you can look for information on various aspects of UNIX in general and Mac OS X in particular.

The Help Viewer

The Help Viewer is a graphical application that you run by clicking Help on the menu bar at the top of the screen. For graphical applications, it is the default help program. In general, the Help Viewer offers the documentation for application programs, not for utilities. This documentation often includes links to useful resources. For example, the help for Terminal includes links to Apple's Web site. These links point to documentation about the UNIX system and utilities.

System Documentation

With the Developer Tools package installed (page 478), the Xcode development environment (page 478) offers access to much of the system programming and UNIX command line documentation as well as provides a general-purpose software development environment.

man: Displaying the System Manual

The character-based man utility displays pages, known as man pages, from the system documentation. This documentation is useful when you know which utility you want to use but have forgotten exactly how to use it. You can also refer to the man pages to get more information about specific topics or to determine which features are available under Mac OS X. Because the descriptions in the system documentation are often terse, they are most helpful when you already understand basically what a utility does.

Because man is a character-based utility, you must run it from the command line. Nicely formatted versions of the man pages are available from within Xcode (page 478).

To find out more about a utility, give the command man, followed by the name of the utility. Figure 2-6 shows the command man who displaying information about the who utility.

Figure 2-6. The man utility displaying information about who

The command man man displays information on man. The man utility automatically sends the output through a pager, usually less (page 42), which allows you to view a file one screen at a time. When you access a manual page in this way, less displays a prompt (:) at the bottom of the screen after each screen of text and waits for you to request another screen by pressing the SPACE bar. Pressing h (help) displays a list of the less commands that you can use. Pressing q (quit) stops less and displays a shell prompt. You can search for topics covered by man pages by using the apropos utility (page 59, or give the command man apropos).

The Mac OS X system manual and the man pages are divided into ten sections. Each section describes related tools:

  1. User Commands

  2. System Calls

  3. Subroutines

  4. Devices

  5. File Formats

  6. Games

  7. Miscellaneous

  8. System Administration

  9. Kernel Interfaces

N. Tcl/Tk commands

This layout closely mimics the way the set of UNIX manuals has always been divided. Unless you specify a manual section, man displays the earliest occurrence in the manual of the word you specify on the command line. Most users find the information they need in sections 1, 6, and 7; programmers and system administrators frequently need to consult the other sections. Some man pages are installed only with the Developer Tools package (page 478).

In some cases there are manual entries for different tools with the same name. For example, the following command displays the manual page for the write utility (page 63) from section 1 of the system manual:

$ man write

To see the manual page for the write() system call from section 2, enter

$ man 2 write

This command instructs man to look only in section 2 for the manual page. Use the a option (see the adjacent tip) to view all the man pages for a given subject (press q to move to the next section); use man a write to view all the man pages for write.

Tip: Options

An option modifies the way a utility or command works. Options are specified as one or more letters that are preceded by one or two hyphens (there are exceptions). The option appears following the name of the utility you are calling and a SPACE. Any other arguments (page 921) to the command follow the option and a SPACE. For more information refer to "Options" on page 115.

Tip: man and info display different information

The info utility displays more complete and up-to-date information on GNU utilities than does man. When a man page displays abbreviated information on a utility that is covered by info, the man page refers you to info. The man utility frequently displays the only information available on non-GNU utilities; when info displays information on non-GNU utilities, it is frequently a copy of the man page.

info: Displaying Information About Utilities

The character-based info utility is a menu-based hypertext system developed by the GNU Project and distributed with Mac OS X. As with man, you must run info from a terminal emulator window (page 24). The info utility includes a tutorial on itself (give the command info info or go to www.gnu.org/software/texinfo/manual/info) and documentation on many Mac OS X shells, utilities, and programs developed by the GNU Project (page 3). Figure 2-7 shows the screen that info displays when you give the command info.

Figure 2-7. The first screen that info displays

Because the information on this screen is drawn from an editable file, your display may differ. From the initial info screen you can press

  • h to go through an interactive tutorial on info

  • ? to list info commands

  • SPACE to scroll through the menu of items you can get information on

  • m, followed by the name of the menu item you want to access

  • q to quit

The notation info uses to describe keyboard keys may not be familiar to you. The notation C-h is the same as CONTROL-H. Similarly M-x means hold down the META or ALT key and press x. Standard Mac keyboards do not have a META or ALT key. The "Activating the META key" tip explains how to set up the OPTION key to perform the same functions as the META key. For more information refer to "Keys: Notation and Use" on page 208.

Tip: Activating the META key

In the Terminal utility, you can make the OPTION (or ALT) key work as the META key. From the Terminal utility File menu, select Window Settings to display the Terminal Inspector window (Figure 2-8). This window provides a drop-down menu of properties that you can change. Select Keyboard, check the box labeled Use option key as meta key, and click Use Settings as Defaults. This procedure causes the OPTION key (on Mac keyboards) or the ALT key (on PC keyboards) to function as the META key while you are using the Terminal utility.

Figure 2-8. Terminal Inspector window

[View full size image]

After giving the command info, press the SPACE bar a few times to scroll the display. Figure 2-9 (next page) shows the first screen that info displays. The asterisk at the left end of the line means that this entry is the beginning of a menu item. Following the asterisk is the name of the menu item, a colon, the name of the package (in parentheses) to which the menu item belongs, other information, and a description of the item (on the right).

Figure 2-9. The screen that info displays after you press the SPACE bar a few times

info menu items

Each menu item is an active link to the info page that describes the item. To jump to that page, move the cursor to the line containing the menu item and press RETURN. Alternatively you can type the name of the menu item in a menu command to view the information. For example, to display information on bash, give the command m bash followed by a RETURN. When you type m (for menu), the cursor moves to the bottom line of the screen and displays Menu item:. Typing bash displays bash on that line, and pressing RETURN displays the bash info page.

Figure 2-10 shows the top node of information on bash. A node is one group of information that you can scroll through with the SPACE bar. To display the next node, press n. Press p to display the previous node. You can always press d to display the initial menu, shown in Figure 2-7.

Figure 2-10. The info page on the bash utility

As you read this book and learn about new utilities, you may want to use man or info to find out more about those utilities. If you can print PostScript documents, you can print a manual page with the man utility using the t option (for example, man t cat | lpr prints information about the cat utility). The Linux Documentation Project (www.tldp.org) has a large collection of documentation on UNIX utilities, much of which is applicable to Mac OS X.

As with the man pages, much of the info documentation is available from within Xcode (page 478).

Using the Internet to Get Help

The Internet provides many helpful sites; see Appendix B for a list. Aside from visiting sites that carry various forms of documentation, you can enter an error message that you are having a problem with in a search engine such as Google (www.google.com). Enclose the error message within double quotation marks to improve the quality of the results. You will likely find a post concerning the problem and ways to solve it. See Figure 2-11.

Figure 2-11. Google reporting on an error message

[View full size image]

The Apple Web site

The Apple Web site is a rich source of information. Following is a list of some locations that may be of interest:

  • Manuals for Apple operating systems and products are available at www.info.apple.com/support/manuals.html.

  • General support material is available at www.apple.com/support.

  • Apple support forums are online discussions about any Apple-related issues. Forums are dedicated to specific applications or OS versions, specific Macintosh models, and so on. Go to discussions.info.apple.com to browse through the forums.

  • Mailing lists are hosted by Apple; see lists.apple.com for a list of mailing lists.

  • Even if you are not using Mac OS X Server, the documentation for Server has a lot of material on command line usage.

GNU

GNU makes many of its manuals available at www.gnu.org/manual. In addition, go to the GNU home page (www.gnu.org) for more documentation and other GNU resources. Many of the GNU pages and resources are available in a wide variety of languages. Mac OS X uses a number of GNU utilities, and many more are available for download.


More About Logging In

The most important options that affect logging in are set from the System Preferences Accounts pane (Figure 2-12). At the bottom of the list of accounts is a picture of a house labeled Login Options. Click this icon to disable the automatic login feature or to select between the List of users and the Name and password text boxes method of logging in. Under Mac OS X Server, the default setup provides name and password text boxes and always requires a user to log in.

Figure 2-12. System Preferences Accounts pane

[View full size image]

What to Do if You Cannot Log In

When you enter your username or password incorrectly, the system shakes the login window from side to side after you enter both your username and password. The shaking effect is supposed to look like someone shaking his head "no." This behavior tells you that you have entered either the username or the password incorrectly, or that they are not valid. The behavior does not differentiate between an unacceptable login name and an unacceptable password, in an effort to discourage unauthorized people from guessing names and passwords to gain access to the system. The additional security of not disclosing the existence of a specific account is one of the advantages that the name and password text boxes method of logging in has over the list of users method. If you cannot log in check the following:

  • The login name and password are case sensitive.

    Make sure that the CAPS LOCK key is off and that you enter your name and password exactly as specified or as you set them up. If the CAPS LOCK key is on, the login window utility helpfully displays an up-arrow graphic to warn you.

  • Make sure your login name is valid.

    Another reason the login/password combination may not be valid is that you have not been set up as a user. If you are picking your account from a list, only available accounts are listed, so you know that your account has been set up. Otherwise, check with the system administrator. Mac OS X allows you to use either your long or short username (the long name is generally your name; the short name is a lowercase name that is set up when your account is created).

Refer to "Changing Your Password" on page 36 when you want to change your password.

Logging In on the Text Console

The name and password text box method of logging in (Figure 2-2, page 21) offers access to a nongraphical login called the text console that provides an interface similar to a traditional text terminal. To use the text console, enter the string >console in the Name text box and press RETURN. This special login string does not require a password. When you press RETURN OS X displays a blank screen with a traditional, textual UNIX Login: prompt. Enter your username and password in response to the prompts as described in "Textual Login" on page 21.

Once you log in in this manner, you are working with the shell and a command line interface (page 23)you have no access to graphical features. You can use the text console as a fail-safe login because it bypasses the graphical environment, including Finder. Frequently you can repair or replace (from backups) corrupt files in this environment when you cannot log in to a graphical environment. The text console is the same environment that OS X provides in single-user mode (page 437).

Logging Out

To log out from a graphical interface, select Log Out username (username is replaced by your long username) from the Apple menu. From a character-based interface, press CONTROL-D or give the command exit in response to the shell prompt. Exiting from a shell does not end a graphical session; it just exits from the shell you are working with. For example, exiting from the shell that Terminal provides closes the Terminal window.

Changing Your Password

If someone else assigned you a password, it is a good idea to give yourself a new one. A good password is seven or eight characters long and contains a combination of numbers, uppercase and lowercase letters, and punctuation characters. Avoid using control characters (such as CONTROL-H) because they may have a special meaning to the system, making it impossible for you to log in. Do not use names, words from English or other languages, or other familiar words that someone can easily guess.

For security reasons none of the passwords you enter is ever displayed by any utility.

Security: Protect your password

Do not allow someone to find out your password: Do not put your password in a file that is not encrypted, allow someone to watch you type your password, give it to someone you do not know (a system administrator never needs to know your password), or write it down.

Security: Choose a password that is difficult to guess

Do not use phone numbers, names of pets or kids, birthdays, words from a dictionary (not even a foreign language), and so forth. Do not use permutations of these items.

Security: Differentiate between important and less important passwords

It is important to differentiate between important and less important passwords. For example, Web site passwords for blogs or download access are not very important; it is acceptable to use the same password for these types of sites. However, your login, mail server, and bank account Web site passwords are critical: Never use these passwords for an unimportant Web site.

Your password should meet the following criteria to be relatively secure:

  • It should be at least six characters long.

  • It should not be a word in a dictionary of any language, no matter how seemingly obscure.

  • It should not be the name of a person, place, pet, or other thing that might be discovered easily.

  • It should contain at least two letters and one digit.

  • It should not be your login name, the reverse of your login name, or your login name shifted by one or more characters.

  • If you are changing your password, the new password should differ from the old one by at least three characters. Changing the case of a character does not make it count as a different character.

When you successfully change your password, you change the way you log in. If you forget your password, Superuser can change it and tell you your new password.

Changing Your Password Graphically

To change your password in a GUI, open System Preferences, go to the Accounts pane, select your account from the list on the left, and click Change Password. You must enter your new password twice before clicking Change Password or the system will not change it. When you change your password this way, you can set a password hint to remind you of your password. However, setting password hints is a poor security practice; it is better to pick a password you can remember.

Changing Your Password Textually

You can also change your password by giving the command passwd from the command line; using the passwd utility changes your login password, but does not change your Keychain password. (The Keychain password is used by various graphical applications. You can change the Keychain password using the Keychain Access application.)

The first item passwd asks you for is your old password. This password is verified to ensure that an unauthorized user is not trying to alter your password. Next the system requests the new password.

After you enter your new password, the system asks you to retype it to ensure you did not make a mistake when you entered it the first time. If the new password is the same both times you enter it, your password is changed. If the passwords differ, it means that you made an error in one of them, and the system displays an error message:

Mismatch; try again, EOF to quit.


Chapter Summary

The system administrator is responsible for maintaining the system. On a single-user system, you are the system administrator. On a small, multiuser system, you or another user acts as the system administrator, or this job may be shared. On a large, multiuser system or network of systems, there is frequently a full-time system administrator. When extra privileges are required to perform certain system tasks, the system administrator logs in as the root user by entering the username root and the root password or uses the sudo utility to run commands directly. While logged in in this manner, this user is called Superuser or administrator. On a multiuser system, several trusted users may be given the root password. Do not work as Superuser as a matter of course. When you have to do something that requires Superuser privileges, work as Superuser for only as long as you need to; revert to working as yourself as soon as possible.

A terminal emulator, such as the Terminal utility, allows you to use the Mac OS X command line interface from a graphical environment. You can use a terminal emulator to launch both textual and graphical programs.

The man utility provides online documentation on system utilities. This utility is helpful to new users as well as to experienced users who must often delve into the system documentation for information on the fine points of a utility's behavior. The info utility helps the beginner and expert alike. This utility includes a tutorial on its use and documentation on many utilities.

As with many operating systems, your access to a Mac OS X system is authorized when you log in. You can change your password at any time. Choose a password that is difficult to guess and that conforms to the criteria imposed by the utility that changes your password.


Exercises

1. | In either a graphical or textual environment, when you provide a username and password to log in and your login is unsuccessful, OS X does not indicate whether the username, the password, or both are invalid. Why does the system not tell you this information?
2. | Give three examples of poor password choices. What is wrong with each? Include one that is too short.
3. | Is fido an acceptable password? Give several reasons why or why not.
4. | What would you do if you could not log in?


Advanced Exercises

5. | How many man pages are in the Devices subsection of the system manual? (Hint: Devices is a subsection of Special Files.)
6. | The example on page 29 shows that man pages for write appear in sections 1 and 2 of the system manual. Explain how you can use man to determine which sections of the system manual contain a manual page with a given name.
7. | How would you find out which OS X utilities create and work with archive files?


Chapter 3. The Mac OS X Utilities

Previous Page
Next Page

3. The Mac OS X Utilities

IN THIS CHAPTER

Special Characters | 40
Basic Utilities | 41
less: Displays a Text File One Screen at a Time | 42
Working with Files | 43
lpr: Prints a File | 45
| (Pipe): Communicates Between Processes | 51
Compressing and Archiving Files | 54
Locating Commands | 58
apropos: Searches for a Keyword | 59
Obtaining User and System Information | 60

When the first UNIX systems were introduced, and for a long time thereafter, UNIX did not have a graphical interface: It ran on character-based terminals only. All the tools ran from the command line. The Mac OS X GUI is important, of course, but many peopleespecially system administratorsrun many command line programs. In a lot of cases, command line utilities are faster, more powerful, or more complete than their GUI counterparts. Sometimes there is no GUI counterpart for a text-based utility, and some people just prefer the hands-on feeling of the command line.

When you work with a command line interface, you are working with the shell (Chapters 5, 8, and 13). Before you start working with the shell, it is important to understand something about the characters that are special to the shell, so this chapter starts with a discussion of special characters. The chapter then describes six basic utilities: ls, cat, rm, open, less, and hostname. It continues by describing several other file manipulation utilities as well as utilities that find out who is logged in; that communicate with other users; that print, compress, and decompress files; and that pack and unpack archived files.

Tip: Run these utilities from a command line

This chapter describes command line, or text-based, utilities. You can experiment with these utilities from the Terminal utility (page 24), a terminal emulator within a GUI.


Special Characters

Special characters, which have a special meaning to the shell, are discussed in "Filename Generation/Pathname Expansion" on page 133. These characters are mentioned here so that you can avoid accidentally using them as regular characters until you understand how the shell interprets them. For example, avoid using any of these characters in a filename (even though emacs and some other programs do) until you learn how to quote them (next). The standard special characters are

& ; | * ? ' " ` [ ] ( ) $ < > { } ^ # / \ % ! ~ +

Although not considered special characters, RETURN, SPACE, and TAB also have special meanings to the shell. RETURN usually ends a command line and initiates execution of a command. The SPACE and TAB characters separate elements on the command line and are collectively known as whitespace or blanks.

Quoting special characters

If you need to use as a regular character one of the characters that has a special meaning to the shell, you can quote (or escape) it. When you quote a special character, you keep the shell from giving it special meaning. The shell treats a quoted special character as a regular character. However, a slash (/) is always a separator in a pathname, even when you quote it.

To quote a character, precede it with a backslash (\). When you have two or more special characters together, you must precede each with a backslash (for example, enter ** as \*\*). You can quote a backslash just as you would quote any other special characterby preceding it with a backslash (\\).

Another way of quoting special characters is to enclose them between single quotation marks, as in '**'. You can quote many special and regular characters between a pair of single quotation marks, as in 'This is a special character: >'. The regular characters remain regular, and the shell also interprets the special characters as regular characters.

The only way to quote the erase character (CONTROL-H), the line kill character (CONTROL-U), and other control characters (try CONTROL-M) is by preceding any one with a CONTROL-V. Single quotation marks and backslashes do not work. Try the following:

$ echo 'xxxxxx CONTROL-U'
$ echo 'xxxxxx CONTROL-V CONTROL-U'


Basic Utilities

One of the important advantages of Mac OS X is that it comes with hundreds of utilities that perform myriad functions. You will use utilities whenever you use Mac OS X, whether you access them directly by name from the command line or indirectly from a menu or icon. The following sections discuss some of the most basic and important utilities; these utilities are available from a character-based interface. Several of the more important utilities are also available from a GUI, and some are available only from a GUI.

Folder

The term directory is used extensively in the next sections. A directory is a resource that can hold files. In the Mac OS X graphical interface and in some other operating systems, a directory is referred to as a folder, which is a good analogy: A directory is a folder that can hold files.

Caution: In this chapter you work in your home directory

When you log in on the system, you are working in a directory that is called your home directory. In this chapter that is the only directory you use: All the files you create in this chapter are in your home directory.

ls: Lists the Names of Files

Using the editor of your choice (you can use a GUI editor or experiment with vim [tutorial on page 147] or emacs [tutorial on page 201]), create a small file named practice in your home directory. (In the standard Apple file dialog, the picture of a house with your name next to it represents your home directory.) After exiting from the editor, you can use the ls (list) utility (from the command linefor example, from Terminal) to display a list of the names of the files in the directory. The first command in Figure 3-1 shows ls listing the name of the practice file. (You may also see files that the system or a program created automatically.) Subsequent commands in Figure 3-1 display the contents of the file and remove the file. These commands are described next.

Figure 3-1. Using ls, cat, and rm on the file named practice
$ ls
practice
$ cat practice
This is a small file that I created
with a text editor.
$ rm practice
$ ls
$ cat practice
cat: practice: No such file or directory
$

cat: Displays a Text File

The cat utility displays the contents of a text file. The name of the command is derived from catenate, which means to join together one after the other. (Figure 5-8 on page 124 shows how to use cat to string together the contents of three files.)

A convenient way to display the contents of a file to the screen is by giving the command cat, followed by a SPACE and the name of a file. Figure 3-1 shows cat displaying the contents of practice. This figure shows the difference between the ls and cat utilities: The ls utility displays the name of a file, whereas cat displays the contents of a file.

rm: Deletes a File

The rm (remove) utility deletes a file. Figure 3-1 shows rm deleting the file named practice. After rm deletes the file, ls and cat show that practice is no longer in the directory. The ls utility does not list its filename, and cat says that it cannot open the file. Use rm carefully.

Tip: A safer way of removing files

You can use the interactive form of rm to make sure that you delete only the file(s) you intend to delete. When you follow rm with the -i option (see the tip on page 29) and the name of the file you want to delete, rm displays the name of the file and asks you to respond with y (yes) or n (no) before it deletes the file.

$ rm -i toollist
rm: remove 'toollist'? y

Optional: You can create an alias (page 311) for rm i and put it in your startup file (page 77) so that rm always runs in interactive mode.

By default rm does not act according to the Apple Human Interface Guidelines, which say that an operation should either be reversible or ask for confirmation. In general, UNIX utilities do not ask you whether you are sure of what you are doing.

open: Opens a File

The open utility launches a graphical application. For example, the command open memo.rtf starts TextEdit and loads the file memo.rtf. The application runs independent of the shell, which displays another prompt immediately. On a traditional UNIX system, the open utility would be difficult to implement, but Mac OS X provides a way for the system to tell, for many files, which application is best suited to opening them. For more information refer to "File Attributes" on page 95. Refer to page 803 for more information on open.

less: Displays a Text File One Screen at a Time

When you want to view a file that is longer than one screen, you can use the less utility. This utility pauses after displaying a screen of text. Because it shows one page at a time, it is called a pager. While you are using less, you can press h to display a Help screen that lists commands you can use while paging through a file. Give the command less practice in place of the cat command in Figure 3-1 to see how this command works. Use the command less /etc/services instead if you want to experiment with a longer file. Refer to page 768 for more information on less.

Tip: Filename completion

After you enter one or more letters of a filename on a command line, press TAB and the shell completes as much of the filename as it can. When only one filename starts with the characters you entered, the shell completes the filename and places a SPACE after it. You can keep typing or you can press RETURN to execute the command at this point. When the characters you entered do not uniquely identify a filename, the shell completes what it can and waits for more input. When pressing TAB does not change the display, you can press TAB again (Bourne Again Shell, page 306) or CONTROL-D (TC Shell, page 348) to display a list of possible completions.

hostname: Displays Your System Name

The hostname utility displays the name of the system you are working on. Use this utility if you are not sure that you are logged in on the right machine.

$ hostname
bravo.local


Working with Files

This section describes utilities that copy, move, print, search through, display, sort, and compare files.

Caution: Resource forks: some files have additional data

All files, except for empty files, contain data. Unlike traditional UNIX files, Mac OS X files can contain additional information about a file. This information is kept in forks. The data is kept in the data fork. Additional information is kept in the resource fork. The resource fork can be empty or it can hold important information. For more information refer to "Resource forks" on page 93.

Under Mac OS X 10.4 and later, all of the utilities in this section, and almost all other utilities that come with Mac OS X, recognize and work with both data and resource forks. An exception occurs when you redirect the output of a utility. Refer to "| (Pipe): Communicates Between Processes" on page 51 for more information.

Under Mac OS X 10.3 and earlier, many utilities recognize only the data fork of a file. If you are not sure if a utility respects resource forks, make a copy of the file using the Finder, which always respects resource forks, before modifying a file with the suspect utility.

cp: Copies a File

The cp (copy) utility (Figure 3-2, next page) makes a copy of a file. This utility can copy any file, including text and executable program (binary) files. You can use cp to make a backup copy of a file or a copy to experiment with. Under Mac OS X 10.4 and later, cp copies resource forks; under version 10.3 and earlier, it does not. Refer to the preceding tip for information on resource forks. See ditto (next) if you need to copy resource forks under earlier versions of OS X.

Figure 3-2. cp copies a file
$ ls
memo
$ cp memo memo.copy
$ ls
memo memo.copy

A cp command line uses the following syntax to specify source and destination files:

cp source-file destination-file

The source-file is the name of the file that cp will copy. The destination-file is the name that cp assigns to the resulting (new) copy of the file.

Tip: cp can destroy a file

If the destination-file exists before you give a cp command, cp overwrites it. Because cp overwrites (and destroys the contents of) an existing destination-file without warning, you must take care not to cause cp to overwrite a file that you need. The cp -i (interactive) option (see page 29 for a tip on options) prompts you before it overwrites a file.

The following example assumes that the file named orange.2 exists before you give the cp command. The user answers y to overwrite the file.

$ cp -i orange orange.2
overwrite orange.2? (y/n [n]) y

The command line shown in Figure 3-2 copies the file named memo to memo.copy. The period is part of the filenamejust another character. The initial ls command shows that memo is the only file in the directory. After the cp command, the second ls shows both files, memo and memo.copy, in the directory.

Sometimes it is useful to incorporate the date into the name of a copy of a file. The following example includes the date January 30 (0130):

$ cp memo memo.0130

Although it has no significance to the system, the date can help you find a version of a file that you created on a certain date. The date can also help you avoid overwriting existing files by providing a unique filename each day. Refer to "Filenames" on page 74.

Use scp (page 832) or ftp (page 738) when you need to copy a file from one system to another on a network.

ditto: Copies Files and Directories

The ditto utility is similar to cp but is not available on traditional UNIX systems. Without any options, ditto copies files as well as directories and their contents. It can also create archives (page 921) of files or extract files from archives. For example, the following example makes a dated copy of the Documents directory:

$ ditto Documents Documents.0130

This command copies the Documents directory hierarchy, preserving the resource forks of the files it copies. If you are running Mac OS X 10.3 or earlier, you must specify the -rsrc option if you want to copy resource forks. This option is not required under 10.4 and later. For more information on options, refer to page 115.

Unlike cp under Mac OS X 10.3 and earlier, the ditto utility not only copies the data a file contains (as does cp), but also copies resource forks (see the tip on page 43). Refer to page 715 for more information on ditto.

mv: Changes the Name of a File

The mv (move) utility renames a file without making a copy of it. The mv command line specifies an existing file and a new filename using the same syntax as cp:

mv existing-filename new-filename

The command line in Figure 3-3 changes the name of the file memo to memo.0130. The initial ls command shows that memo is the only file in the directory. After you give the mv command, memo.0130 is still the only file in the directory. Compare this result to the earlier cp example.

Figure 3-3. mv renames a file
$ ls
memo
$ mv memo memo.0130
$ ls
memo.0130

The mv utility can be used for more than changing the name of a file. Refer to "mv, cp: Moves or Copies a File" on page 84.

Using the mv utility is analogous to using the Finder to rename a file or move a file from one folder to another. This utility also supports resource forks. Refer to page 792 for more information on mv.

lpr: Prints a File

The lpr (line printer) utility places one or more files in a print queue for printing. Mac OS X provides print queues so that only one job gets printed on a given printer at a time. Such a queue allows several people or jobs to send output simultaneously to a single printer with the expected results. On systems with access to more than one printer, use lpstat -p to display a list of available printers. Use the -P option to instruct lpr to place the file in the queue for a specific printer, including one that is connected to another system on the network. The following command prints the file named report:

$ lpr report

Because this command does not specify a printer, the output goes to the default printer, which is the printer when you have only one.

The next command line prints the same file on the printer named mailroom:

$ lpr -Pmailroom report

You can see which jobs are in the print queue by using the lpq utility:

$ lpq
lp is ready and printing
Rank  Owner   Job Files                  Total Size
active alex    86 (standard input)         954061 bytes

In this example, Alex has one job that is being printed; no other jobs are in the queue. You can use the job number, 86 in this case, with the lprm utility to remove the job from the print queue and stop it from printing:

$ lprm 86

You can send more than one file to the printer with a single command. The following command line prints three files on the printer named laser1:

$ lpr -Plaser1 05.txt 108.txt 12.txt

Refer to page 774 for more information on lpr.

grep: Searches for a String

The grep[1] utility searches through one or more files to see whether any contain a specified string of characters. It does not change the file it searches but simply displays each line that contains the string.

[1] Originally the name grep was a play on an edan original UNIX editor, available on Mac OS Xcommand: g/re/p. In this command the g stands for global, re is a regular expression delimited by slashes, and p means print.

The grep command in Figure 3-4 searches through the file memo for lines that contain the string credit and displays a single line that meets this criterion. If memo contained such words as discredit, creditor, or accreditation, grep would have displayed those lines as well because they contain the string it was searching for. The -w option causes grep to match only whole words. You do not need to enclose the search string in single quotation marks, but doing so allows you to put SPACEs and special characters in it.

Figure 3-4. grep searches for a string
$ cat memo

Helen:

In our meeting on June 6 we
discussed the issue of credit.
Have you had any further thoughts
about it?
                       Alex
$ grep 'credit' memo
discussed the issue of credit.

The grep utility can do much more than search for a simple string in a single file. Refer to grep on page 751 and in Appendix A, "Regular Expressions," for more information.

head: Displays the Beginning of a File

By default the head utility displays the first ten lines of a file. You can use head to help you remember what a particular file contains. If you have a file named months that lists the 12 months of the year in calendar order, one to a line, head displays Jan through Oct (Figure 3-5).

Figure 3-5. head displays the first lines of a file
$ cat months
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec

$ head months
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct

This utility can display any number of lines, so you can use it to look at only the first line of a file, at a full screen, or even more. To specify the number of lines head displays, include a hyphen followed by the number of lines in the head command. For example, the following command displays only the first line of months:

$ head -1 months
Jan

Refer to page 759 for more information on head.

tail: Displays the End of a File

The tail utility is similar to head but by default displays the last ten lines of a file. Depending on how you invoke it, the tail utility can display fewer or more than ten lines, use a count of characters rather than lines to display parts of a file, and display lines being added to a file that is changing. The following command causes tail to display the last five lines, Aug through Dec, of the months file shown in Figure 3-5:

$ tail -5 months
Aug
Sep
Oct
Nov
Dec

You can monitor lines as they are added to the end of the file named logfile with the following command:

$ tail -f logfile

Press the interrupt key (usually CONTROL-C) to stop tail and display the shell prompt. Refer to page 859 for more information on tail.

sort: Displays a File in Order

The sort utility displays the contents of a file in order by lines but does not change the original file. If a file named days contains the name of each day of the week in calendar order, each on a separate line, sort displays the file in alphabetical order (Figure 3-6).

Figure 3-6. sort displays a file in order
$ cat days
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
Sunday
$ sort days
Friday
Monday
Saturday
Sunday
Thursday
Tuesday
Wednesday

The sort utility is useful for putting lists in order. The -u option generates a sorted list in which each line is unique (no duplicates). The -n option puts a list of numbers in order. Refer to page 837 for more information on sort.

uniq: Removes Duplicate Lines from a File

The uniq (unique) utility displays a file, skipping adjacent duplicate lines, but does not change the original file. If a file contains a list of names and has two successive entries for the same person, uniq skips the extra line (Figure 3-7).

Figure 3-7. uniq removes duplicate lines
$ cat dups
Cathy
Fred
Joe
John
Mary
Mary
Paula
$ uniq dups
Cathy
Fred
Joe
John
Mary
Paula

If a file is sorted before it is processed by uniq, this utility ensures that no two lines in the file are the same. (Of course, sort can do that all by itself with the -u option.) Refer to page 885 for more information on uniq.

diff: Compares Two Files

The diff (difference) utility compares two files and displays a list of the differences between them. This utility does not change either file and is useful when you want to compare two versions of a letter or a report or two versions of the source code for a program.

The diff utility with the -u (unified output format) option first displays two lines indicating which of the files you are comparing will be denoted by a plus sign (+) and which by a minus sign (-). In Figure 3-8, a minus sign indicates the colors.1 file; a plus sign, the colors.2 file.

Figure 3-8. diff displaying the unified output format
$ diff -u colors.1 colors.2
--- colors.1    Fri Nov 25 15:45:32 2005
+++ colors.2    Fri Nov 25 15:24:46 2005
@@ -1,6 +1,5 @@
 red
+blue
 green
 yellow
-pink
-purple
 orange

The diff -u command breaks long, multiline text into hunks. Each hunk is preceded by a line starting and ending with two at signs (@@). This hunk identifier indicates the starting line number and the number of lines from each file for this hunk. In Figure 3-8, this line indicates that the hunk covers the section of the colors.1 file (indicated by a minus sign) from the first line and continuing for six lines (for a total of seven lines). Similarly the +1,5 indicates that the hunk covers colors.2 from the first line through five subsequent lines.

Following these header lines, diff -u displays each line of text with a leading minus sign, plus sign, or nothing. A leading minus sign indicates that the line occurs only in the file denoted by the minus sign. A leading plus sign indicates that the line comes from the file denoted by the plus sign. A line that begins with neither a plus sign nor a minus sign occurs in both files at the same location. Refer to page 707 for more information on diff.

file: Tests the Contents of a File

You can use the file utility to learn about the contents of any file on a Mac OS X system without having to open and examine the file yourself. In the following example, file reports that letter_e.bz2 contains data that was compressed by the bzip2 utility (page 54):

$ file letter_e.bz2
letter_e.bz2: bzip2 compressed data, block size = 900k

Next file reports on two more files:

$ file memo picture.jpg
memo:        ASCII text
picture.jpg: JPEG image data, JFIF standard 1.01

The file utility ignores creator codes and Macintosh file types (both on page 96); it just looks at the contents of a file (the data fork; see the tip on page 43). Refer to page 726 for more information on file.

| (Pipe): Communicates Between Processes

Because pipes are integral to the functioning of a Mac OS X system, they are introduced here for use in examples. Pipes are covered in detail on page 128.

A process is the execution of a command by Mac OS X (page 119). Communication between processes is one of the hallmarks of UNIX and UNIX-like systems. A pipe (written as a vertical bar, |, on the command line and appearing as a solid or broken vertical line on a keyboard) provides the simplest form of this kind of communication. Simply put, a pipe takes the output of one utility and sends that output as input to another utility. Using UNIX terminology, a pipe takes standard output of one process and redirects it to become standard input of another process. (For more information refer to "Standard Input and Standard Output" on page 120.) Most of what a process displays on the screen is sent to standard output. If you do not redirect it, this output appears on the screen. Using a pipe, you can redirect the output so that it becomes instead standard input of another utility. A utility such as head can take its input from a file whose name you specify on the command line following the word head, or it can take its input from standard input. For example, you can give the command shown in Figure 3-5 on page 47 as follows:

$ cat months | head
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct

The next command displays the number of files in a directory. The wc (word count) utility with the -w option displays the number of words in its standard input or in a file you specify on the command line:

$ ls | wc -w
      14

You can use a pipe to send output of a program to the printer:

$ tail months | lpr

Caution: Pipes do not work with resource forks

Pipes work with the data fork of a file only;they do not work with resource forks (page 43). Although this limitation is probably not an issue now, it may become more important as you read on. For more information see the "Redirection does not support resource forks" tip on page 94.


Three More Utilities

The echo and date utilities are two of the most frequently used from the large collection of Mac OS X utilities. The script utility records part of a session in a file.

echo: Displays Text

The echo utility copies anything you put on the command line after echo to the screen. Some examples are shown in Figure 3-9. The last example shows what the shell does with an unquoted asterisk (*) on the command line. It expands the asterisk into a list of filenames in the directory.

Figure 3-9. echo copies the command line (but not the word echo) to the screen
$ ls
memo  memo.0714  practice
$ echo Hi
Hi
$ echo This is a sentence.
This is a sentence.
$ echo star: *
star: memo memo.0714 practice

The echo utility is a good tool for learning about the shell and other OS X programs. Some examples on page 134 use echo to illustrate how special characters, such as the asterisk, work. Throughout Chapters 8 and 13, echo helps explain how shell variables work and how you can send messages from shell scripts to the screen. Refer to page 720 for more information on echo.

date: Displays the Date and Time

The date utility displays the current date and time:

$ date
Mon Mar 13 08:00:38 PST 2006

The following example shows how you can choose the format and select the contents of the output of date:

$ date +"%A %B %d"
Monday March 13

Refer to page 701 for more information on date.

script: Records a Shell Session

The script utility records all or part of a login session, including your input and the system's responses. This utility is useful only from character-based devices, such as a terminal or a terminal emulator. It does capture a session with vim; however, because vim uses control characters to position the cursor and display different typefaces, such as bold, the output will be difficult to read and may not be useful. When you cat a file that has captured a vim session, the session quickly passes before your eyes.

By default script captures the session in a file named typescript. To use a different filename, follow the script command with a SPACE and the filename you want to use. To append material to a file, use the -a option after script but before the filename; otherwise script overwrites an existing file. Following is a session being recorded by script:

$ script
Script started, output file is typescript
$ date
Mon Jan  3 14:26:35 CST 2005
$ who am i
alex     ttyp3    Jan  3 14:26
$ apropos msdos
fsck_msdos(8)          - DOS/Windows (FAT) file system consistency check
mount_msdos(8)          - mount an MS-DOS file system
msdos.util(8)           - DOS/Windows (FAT) file system utility
newfs_msdos(8)          - construct a new MS-DOS (FAT) file system
$ exit
exit
Script done, output file is typescript
$

Use the exit command to terminate a script session. You can view the file you created with cat, less, or an editor. Following is the file that was created by the preceding script command:

$ cat typescript
Script started on Mon Jan  3 14:26:34 2005
$ date
Mon Jan  3 14:26:35 CST 2005
$ who am i
alex     ttyp3    Jan  3 14:26
$ apropos msdos
fsck_msdos(8)         - DOS/Windows (FAT) file system consistency check
mount_msdos(8)         - mount an MS-DOS file system
msdos.util(8)          - DOS/Windows (FAT) file system utility
newfs_msdos(8)         - construct a new MS-DOS (FAT) file system
$ exit
exit
Script done on Mon Jan  3 14:28:35 2005
$

If you will be editing the file with vim, emacs, or another editor, you can use TR (translate, page 879) as shown in the following command to eliminate from the typescript file the ^M characters that appear at the ends of lines:

$ cat typescript | tr -d '\r' > typescript.good

The greater than (>) symbol redirects the standard output of tr to the file named transcript.good. For more information refer to "Redirecting Standard Output" on page 122.


Compressing and Archiving Files

Large files use a lot of disk space and take longer than smaller files to transfer from one system to another over a network. If you do not need to look at the contents of a large file very often, you may want to save it on a CD, DVD, or other medium and remove it from the hard disk. If you have a continuing need for the file, retrieving a copy from a CD may be inconvenient. To reduce the amount of disk space you use without removing the file entirely, you can compress the file without losing any of the information it holds. Similarly a single archive of several files packed into a larger file is easier to manipulate, upload, download, and email than multiple files. You may frequently download compressed, archived files from the Internet. The utilities described in this section compress and decompress files and pack and unpack archives.

bzip2: Compresses a File

The bzip2 utility (sources.redhat.com/bzip2) compresses a file by analyzing it and recoding it more efficiently. The new version of the file looks completely different. In fact, because the new file contains many nonprinting characters, you cannot view it directly. The bzip2 utility works particularly well on files that contain a lot of repeated information, such as text and image data, although most image data is already in a compressed format.

The following example shows a boring file. Each of the 8,000 lines of the letter_e file contains 72 e's and a NEWLINE character that marks the end of the line. The file occupies more than half a megabyte of disk storage.

$ ls -l
-rw-rw-r--  1 sam sam 584000 Mar  1 22:31 letter_e

The -l (long) option causes ls to display more information about a file. Here it shows that letter_e is 584,000 bytes long. The --verbose (or -v) option causes bzip2 to report how much it was able to reduce the size of the file. In this case, it shrank the file by 99.99 percent:

$ bzip2 -v letter_e
letter_e: 11680.00:1, 0.001 bits/byte, 99.99% saved, 584000 in, 50 out.

$ ls -l
-rw-rw-r--  1 sam sam 50 Mar  1 22:31 letter_e.bz2

.bz2 filename extension

Now the file is only 50 bytes long. The bzip2 utility also renamed the file, appending .bz2 to its name. This naming convention reminds you that the file is compressed; you would not want to display or print it, for example, without first decompressing it. The bzip2 utility does not change the modification date associated with the file, even though it completely changes the file's contents.

In the following, more realistic example, the file zach.jpg contains a computer graphics image:

$ ls -l
-rw-r--r-- 1 sam sam 33287 Mar  1 22:40 zach.jpg

The bzip2 utility can reduce the size of the file by only 28 percent because the image is already in a compressed format:

$ bzip2 -v zach.jpg
zach.jpg:  1.391:1,  5.749 bits/byte, 28.13% saved, 33287 in, 23922 out.

$ ls -l
-rw-r--r--  1 sam sam 23922 Mar  1 22:40 zach.jpg.bz2

Refer to page 668 for more information on bzip2.

bunzip2 and bzcat: Decompress a File

You can use the bunzip2 utility to restore a file that has been compressed with bzip2:

$ bunzip2 letter_e.bz2
$ ls -l
-rw-rw-r--  1 sam sam 584000 Mar  1 22:31 letter_e

$ bunzip2 zach.jpg.bz2
$ ls -l
-rw-r--r--  1 sam sam  33287 Mar  1 22:40 zach.jpg

The bzcat utility displays a file that has been compressed with bzip2. The equivalent of cat for .bz2 files, bzcat decompresses the compressed data and displays the contents of the decompressed file. Like cat, bzcat does not change the source file. The pipe in the following example redirects the output of bzcat so that instead of being displayed on the screen it becomes input to head, which displays the first two lines of the file:

$ bzcat letter_e.bz2 | head -2
eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

After bzcat is run, the contents of letter_e.bz is unchanged; the file is still stored on the disk in compressed form.

bzip2recover

The bzip2recover utility supports limited data recovery from media errors. Give the command bzip2recover followed by the name of the compressed, corrupted file from which you want to recover data.

gzip: Compresses a File

gunzip and zcat

The gzip (GNU zip) utility is older and less efficient than bzip2. Its flags and operation are very similar to those of bzip2. A file compressed by gzip is marked by a .gz filename extension. Files you download from the Internet are frequently in gzip format. Use gzip, gunzip, and zcat just as you would use bzip2, bunzip2, and bzcat. Refer to page 756 for more information on gzip.

compress

The compress utility can also compress files, albeit not as well as gzip. This utility marks a file it has compressed by adding .Z to its name.

Tip: gzip versus zip

Do not confuse gzip and gunzip with the zip and unzip utilities. These last two are used to pack and unpack zip archives containing several files compressed into a single file that has been imported from or is being exported to a system running Windows. The zip utility constructs a zip archive, whereas unzip unpacks zip archives. The zip and unzip utilities are compatible with PKZIP, a Windows compress and archive program.

tar: Packs and Unpacks Archives

The tar utility performs many functions. Its name is short for tape archive, as its original function was to create and read archive and backup tapes. Today it is used to create a single file (called a tar file or archive) from multiple files or directory hierarchies and to extract files from a tar file. The cpio (page 693), ditto (page 715), and pax (page 809) utilities perform similar functions.

In the following example, the first ls shows the existence and sizes of the files g, b, and d. Next tar uses the -c (create), -v (verbose), and -f (write to or read from a file) options[2] to create an archive named all.tar from these files. Each line of the output from tar lists the name of a file that it is appending to the archive.

[2] Although the original UNIX tar did not use a leading hyphen to indicate an option on the command line, it now accepts hyphens. The GNU tar described here will accept tar commands with or without a leading hyphen. This book uses the hyphen for consistency with most other utilities.

The tar utility adds overhead when it creates an archive. The next command shows that the archive file all.tar occupies about 9,700 bytes, whereas the sum of the sizes of the three files is about 6,000 bytes. This overhead is more appreciable on smaller files, such as the ones in this example.

$ ls -l g b d
-rw-r--r--   1 jenny jenny 1302 Aug 20 14:16 g
-rw-r--r--   1 jenny other 1178 Aug 20 14:16 b
-rw-r--r--   1 jenny jenny 3783 Aug 20 14:17 d

$ tar -cvf all.tar g b d
g
b
d
$ ls -l all.tar
-rw-r--r--   1 jenny jenny 9728 Aug 20 14:17 all.tar
$ tar -tvf all.tar
-rw-r--r-- jenny/jenny    1302 2005-08-20 14:16 g
-rw-r--r-- jenny/other    1178 2005-08-20 14:16 b
-rw-r--r-- jenny/jenny    3783 2005-08-20 14:17 d

The final command in the preceding example uses the -t option to display a table of contents for the archive. Use -x instead of -t to extract files from a tar archive. Omit the -v option if you want tar to do its work silently.

You can use bzip2, compress, or gzip to compress tar files and make them easier to store and handle. Many files you download from the Internet are in one of these formats. Files that have been processed by tar and compressed by bzip2 frequently have a filename extension of .tar.bz2 or .tbz. Those processed by tar and gzip have an extension of .tar.gz or .tz, while files processed by tar and compress use .tar.Z as the extension.

You can unpack a tarred and gzipped file in two steps. (Follow the same procedure if the file was compressed by bzip2, but use bunzip2 instead of gunzip.) The next example shows how to unpack the GNU make utility after it has been downloaded (ftp.gnu.org/pub/gnu/make/make-3.80.tar.gz):

$ ls -l mak*
-rw-rw-r--  1 sam sam 1211924 Jan 20 11:49 make-3.80.tar.gz

$ gunzip mak*
$ ls -l mak*
-rw-rw-r--  1 sam sam 4823040 Jan 20 11:49 make-3.80.tar

$ tar -xvf mak*
make-3.80/
make-3.80/po/
make-3.80/po/Makefile.in.in
...
make-3.80/tests/run_make_tests.pl
make-3.80/tests/test_driver.pl

The first command lists the downloaded tarred and gzipped file: make-3.80.tar.gz (about 1.2 megabytes). The asterisk (*) in the filename matches any characters in any filenames (page 134), so you end up with a list of files whose names begin with mak; in this case there is only one. Using an asterisk saves typing and can improve accuracy with long filenames. The gunzip command decompresses the file and yields make-3.80.tar (no .gz extension), which is about 4.8 megabytes. The tar command creates the make-3.80 directory in the working directory and unpacks the files into it.

$ ls -ld mak*
drwxrwxr-x  8 sam sam    4096 Oct  3  2002 make-3.80
-rw-rw-r--  1 sam sam 4823040 Jan 20 11:49 make-3.80.tar
$ ls -l make-3.80
total 3536
-rw-r--r--  1 sam sam  24687 Oct  3  2002 ABOUT-NLS
-rw-r--r--  1 sam sam   1554 Jul  8  2002 AUTHORS
-rw-r--r--  1 sam sam  18043 Dec 10  1996 COPYING
-rw-r--r--  1 sam sam  32922 Oct  3  2002 ChangeLog
...
-rw-r--r--  1 sam sam  16520 Jan 21  2000 vmsify.c
-rw-r--r--  1 sam sam  16409 Aug  9  2002 vpath.c
drwxrwxr-x  5 sam sam   4096 Oct  3  2002 w32

After tar extracts the files from the archive, the working directory contains two files whose names start with mak: make-3.80.tar and make-3.80. The -d (directory) option causes ls to display only file and directory names, not the contents of directories as it normally does. The final ls command shows the files and directories in the make-3.80 directory. Refer to page 862 for more information on tar.

Caution: tar: the -x option may extract a lot of files

Some tar archives contain many files. Run tar with the -t option and the name of the tar file to list the files in the archive without unpacking them. In some cases you may want to create a new directory (mkdir [page 80]), move the tar file into that directory, and expand it there. That way the unpacked files do not mingle with existing files, and there is no confusion. This strategy also makes it easier to delete the extracted files. Some tar files automatically create a new directory and put the files into it. Refer to the preceding example.

Caution: tar: the -x option can overwrite files

The -x option to tar overwrites a file that has the same filename as a file you are extracting. Follow the suggestion in the preceding caution box to avoid overwriting files.

Optional

You can combine the gunzip and tar commands on one command line with a pipe (|), which redirects the output of gunzip so that it becomes the input to tar:

$ gunzip -c make-3.80.tar.gz | tar -xvf -

The -c option causes gunzip to send its output through the pipe instead of creating a file. Refer to "Pipes" (page 128), gzip (page 756), and tar (page 862) for more information about how this command line works.

A simpler solution is to use the -z option to tar. This option causes tar to call gunzip (or gzip when you are creating an archive) directly and simplifies the preceding command line to

$ tar -xvzf make-3.80.tar.gz

In a similar manner, the -j option calls bzip2 or bunzip2.


Locating Commands

The which and apropos utilities can help you find a command whose name you have forgotten or whose location you do not know. When multiple copies of a utility or program are present, which tells you which copy you will run.

which and type: Locate a Utility

When you give Mac OS X a command, the shell searches a list of directories for a program with that name and runs the first one it finds. This list of directories is called a search path. For information on how to change the search path, refer to "PATH: Where the Shell Looks for Programs" on page 285. If you do not change the search path, the shell searches only a standard set of directories and then stops searching. Other directories on the system may also contain useful utilities, however.

which

The which utility locates utilities by displaying the full pathname of the file for the utility. (Chapter 4 contains more information on pathnames and the structure of the OS X filesystem.) The local system may include several commands that have the same name. When you type the name of a command, the shell searches for the command in your search path and runs the first one it finds. You can find out which copy of the program the shell will run by using which. In the following example, which reports the location of the tar command:

$ which tar
/usr/bin/tar

The which utility can be helpful when a command seems to be working in unexpected ways. By running which, you may discover that you are running a nonstandard version of a tool or a different one than you expected. (Refer to "Important Standard Directories and Files" on page 444 for a list of standard locations for executable files.) For example, if tar is not working properly and you find that you are running /usr/local/bin/tar instead of /usr/bin/tar, you might suspect that the local version is broken.

apropos: Searches for a Keyword

When you do not know the name of the command you need to carry out a particular task, you can use apropos with a keyword to search for it. This utility searches for the keyword in the short description line (the top line) of all man pages and displays those that contain a match. The man utility, when called with the -k (keyword) option, gives you the same output as apropos (it is the same command). The database apropos uses is not on the system as shipped, but is built automatically by the weekly periodic task (page 439). If you turn your machine off periodically (as with a laptop) or if it goes to sleep, the script may not be run. If you get no output from apropos, run the following command as root (or use sudo):

# /usr/libexec/makewhatis.local /usr/share/man:/usr/X11R6/man:/usr/local/man

The following example shows the output of apropos when you call it with the users keyword. The list is sorted alphabetically, with uppercase letters appearing before lowercase letters. The output includes the name of each command, the section of the manual that contains it, and the brief description from the top of the man page. This list includes the utility that you need (users) and identifies other, related tools that you might find useful:

$ apropos users
Net::Netrc(3pm)         - OO interface to users netrc file
crontab(1)               - maintain crontab files for individual users (V3)
ftpcount(1)              - show current number of users for each class
ftpusers(5), ftpchroot(5) - tnftpd(8) access control file
getusershell(3), setusershell(3), endusershell(3) - get valid user shells
last(1)                  - indicate last logins of users and ttys
mesg(1)                  - display (do not display) messages from other users
mysqlaccess(1)           - Create new users to mysql
pdbedit(8)               - manage the SAM database (Database of Samba Users)
snmpusm(1)               - creates and maintains SNMPv3 users on a network entity
sudoers(5)               - list of which users may execute what
users(1)                 - list current users
wall(1)                  - write a message to users

The whatis utility is similar to apropos but finds only complete word matches on the name of the utility.

$ whatis users
users(1)                  - list current users


Obtaining User and System Information

This section covers utilities that display who is using the system, what those users are doing, and how the system is running. To find out who is using the local system, you can employ one of several utilities that vary in the details they provide and the options they support. The oldest utility, who, produces a list of users who are logged in on the local system, the terminal connection each person is using, and the time the person logged in.

The w and finger utilities show more detail, such as each user's full name and the command line each user is running. You can use the finger utility (page 401) to retrieve information about users on remote systems if your computer is attached to a network. Table 3-1 on page 63 summarizes the output of these utilities.

Table 3-1. Comparison of w, who, and finger

Information displayed

w

who

finger

User login name

x

x

x

Terminal line identification (tty)

x

x

x

Login day or time

x

x

Login date and time

x

Idle time

x

x

Which program the user is executing

x

Where the user logged in from

x

x

Full name (or other information from NetInfo)

x

User-supplied vanity information

x

System uptime and load averageuse uptime for this information only

x

who: Lists Users on the System

The who utility displays a list of users who are logged in. In Figure 3-10 the first column of who shows that Zach and Max are logged in. (Zach is logged in both from the console and from two virtual terminals, probably running Terminal sessions.) The second column shows the device that each user's terminal, workstation, or terminal emulator is connected to. The third column shows the date and time the user logged in. An optional fourth column shows in parentheses the name of the system that a remote user logged in from.

Figure 3-10. who lists who is logged in
$ who
zach    console  Mar 21 21:10
zach    ttyp1    Mar 24 01:07
zach    ttyp2    Mar 30 00:32 (localhost)
max     ttyp3    Mar 30 01:11 (bravo)

The information that who displays is useful when you want to communicate with a user at your installation. When the user is logged in, you can use write (page 63) to establish communication immediately. If who does not list the user or if you do not need to communicate immediately, you can send email to that person (page 64).

If the output of who scrolls off the screen, you can redirect the output through a pipe (|, page 51) so that it becomes the input to less, which displays the output one page at a time. You can also use a pipe to redirect the output through grep to look for a specific name.

If you need to find out which terminal you are using or what time you logged in, you can use the command who am i:

$ who am i
max      ttyp3    Mar 30 01:11 (localhost)

finger: Lists Users on the System

Security: finger can be a security risk

By default Mac OS X has remote finger support disabled, so that only local users can use finger to display information about local users. This utility can give information that can help a malicious user break into the system.

You can use finger to display a list of the users who are logged in on the system. In addition to usernames, finger supplies each user's full name along with information about which device the user's terminal is connected to, how recently the user typed something on the keyboard, when the user logged in, and any contact information stored in the user database. See Figure 3-11.

Figure 3-11. finger I: lists who is logged in
$ finger
Login    Name                     TTY  Idle  Login  Time   Office  Phone
max      Max                      p3         Wed    01:11
zach     Zach                     con    8d  Mar 21 21:10
zach     Zach                     p1         Mar 24 01:07
zach     Zach                     p2      1  Wed    00:32

You can also use finger to learn more about a particular individual by specifying more information on the command line. In Figure 3-12, finger displays detailed information about the user named Max. Max is logged in and actively using his terminal. If Max were not actively using his terminal, finger would report how long he had been idle. You also learn from finger that if you want to set up a meeting with Max, you should contact Sam.

Figure 3-12. finger II: lists details about one user
$ finger max
Login: max                              Name: Max
Directory: /Users/max                   Shell: /bin/bash
On since Wed Mar 30 01:11 (CST) on ttyp3 (messages off) from localhost
No Mail.
Plan: Out of town this week, contact Sam if you need to see me.

.plan and .project

Most of the information in Figure 3-12 was collected by finger from system files. The information shown after the heading Plan:, however, was supplied by Max. The finger utility searched for a file named .plan in Max's home directory and displayed its contents. (Filenames that begin with a period, such as .plan, are not normally listed by ls and are not displayed by the Finder; they are called invisible filenames [page 76].) You may find it helpful to create a .plan file for yourself; it can contain any information you choose, such as your typical schedule, interests, phone number, or address. In a similar manner, finger displays the contents of the .project file in your home directory. If Max had not been logged in, finger would have reported the last time he logged in, the last time he read his email, and his plan.

You can use finger to display a user's username. For example, on a system with a user named Helen Simpson, you might know that Helen's last name is Simpson but might not guess that her username is hls. The finger utility, which is not case sensitive, can search for information on Helen using her first or last name. The following commands find the information you seek as well as information on other users whose names are Helen or Simpson.

$ finger HELEN
Login: hls                              Name: Helen Simpson.
...
$ finger simpson
Login: hls                              Name: Helen Simpson.
...

w: Lists Users on the System

The w utility displays a list of the users who are logged in. As discussed in the section on who, the information that w displays is useful when you want to communicate with someone at your installation.

The first column in Figure 3-13 shows that Zach and Max are logged in. The second column shows the designation of the device that each user's terminal is connected to. The third column shows the system that a remote user is logged in from. The fourth column shows the time each user logged in. The fifth column indicates how long each user has been idle (how much time has elapsed since the user pressed a key on the keyboard). The last column shows the command each user is running.

Figure 3-13. The w utility
$ w
 1:19  up 8 days,  4:11, 4 users, load averages: 0.00 0.00 0.00
USER     TTY      FROM              LOGIN@  IDLE WHAT
zach     console  -                21Mar05 8days -
zach     p1       bravo.example.co Thu01       - ssh
zach     p2       -                 0:32       8 bash
max      p3       localhost         1:11       - w

The first line that the w utility displays includes the time of day, the period of time the computer has been running (in days, hours, and minutes), the number of users logged in, and the load average (how busy the system is). The three load average numbers represent the number of jobs waiting to run, averaged over the past 1, 5, and 15 minutes. Table 3-1 compares the w, who, and finger utilities.


Communicating with Other Users

The utilities discussed in this section exchange messages and files with other users either interactively or through email. They perform functions similar to iChat and Mail.

write: Sends a Message

The write utility sends a message to another user who is logged in. When you and another user use write to send messages to each other, you establish two-way communication. Initially a write command (Figure 3-14) displays a banner on the other user's terminal, saying that you are about to send a message.

Figure 3-14. The write utility I
$ write max
Hi Max, are you there? o

The syntax of a write command line is

write username [terminal]

The username is the username of the user you want to communicate with. The terminal is an optional device name that is useful if the user is logged in more than once. You can display the usernames and device names of all users who are logged in on the local system by using who, w, or finger.

To establish two-way communication with another user, you and the other user must each execute write, specifying the other's login name as the username. The write utility then copies text, line by line, from one keyboard/display to the other (Figure 3-15). Sometimes it helps to establish a convention, such as typing o (for "over") when you are ready for the other person to type and typing oo (for "over and out") when you are ready to end the conversation. When you want to stop communicating with the other user, press CONTROL-D at the beginning of a line. Pressing CONTROL-D tells write to quit, displays EOF (end of file) on the other user's terminal, and returns you to the shell. The other user must do the same.

Figure 3-15. The write utility II
$ write max
Hi Max are you there? o
Message from max@bravo.example.com on ttyp3 at 16:23 ...
Yes Zach, I'm here. o

If the Message from ... banner appears on your screen and obscures something you are working on, press CONTROL-L or CONTROL-R to refresh the screen and remove the banner. Then you can clean up, exit from your work, and respond to the person who is writing to you. You just have to remember who is writing to you, because the banner will no longer appear on the screen.

A user logged in on the graphical console is listed as logged in on the console terminal and cannot receive messages from write. However, a user on a terminal, including a Terminal window, can receive messages from write; terminals typically have device names such as ttyp3.

mesg: Denies or Accepts Messages

Give the following command when you do not wish to receive messages from another user:

$ mesg n

If Zach had given this command before Max tried to send him a message, Max would have seen the following message:

$ write zach
write: zach has messages disabled

You can allow messages again by entering mesg y. Give the command mesg by itself to display is y (for "yes, messages are allowed") or is n (for "no, messages are not allowed").


Email

You can use email to communicate with users on your system and, if your installation is part of a network, with other users on the network. If you are connected to the Internet, you can communicate electronically with users around the world.

Email utilities differ from write in that email utilities can send a message when the recipient is not logged in. These utilities can also send the same message to more than one user at a time.

Many mail programs are available for OS X, including the original character-based mail program, Netscape/Mozilla mail, mail through emacs, pine, mutt, and Apple's Mail application.

You can use procmail to make any mail program easier to use and more secure. The procmail program (www.procmail.org) creates and maintains mail servers and mailing lists; preprocesses mail by sorting it into appropriate files and directories; starts various programs, depending on the characteristics of incoming mail; forwards mail; and so on. (Many of procmail's mail-sorting features are available natively in the Mail application.)

Many mail programs assume that mail is being delivered to a spool file on the local system, the traditional UNIX setup. The character-based mail program, for example, does not know about POP serversit just knows to look for a mailbox file in a standard location. By default a Mac OS X system is not set up for use with such mail clients.

Network addresses

If your system is part of a LAN, you can generally send mail to and receive mail from users on other systems on the LAN by using their login names. Someone sending Alex email on the Internet would need to specify his domain name (page 411) along with his login name. Use this address to send the author email:

mgs@sobell.com


Chapter Summary

The utilities introduced in this chapter and Chapter 2 are a small but powerful subset of the utilities available on a Mac OS X system. Because you will use them frequently and because they are integral to the following chapters, it is important that you become comfortable using them.

The utilities listed in Table 3-2 manipulate, display, compare, and print files.

Table 3-2. File utilities

cp

Copies one or more files (page 43)

diff

Displays the differences between two files (page 49)

ditto

Copies files and directories (page 44)

file

Displays information about the contents of a file (page 50)

grep

Searches a file for a string (page 46)

head

Displays the lines at the beginning of a file (page 47)

lpq

Displays a list of jobs in the print queue (page 46)

lpr

Places file(s) in the print queue (page 45)

lprm

Removes a job from the print queue (page 46)

mv

Renames file(s) or moves file(s) to another directory (page 45)

open

Opens a file (page 42)

script

Records a shell session (page 52)

sort

Puts a file in order by lines (page 49)

tail

Displays the lines at the end of a file (page 48)

uniq

Displays the contents of a file, skipping successive duplicate lines (page 49)

To reduce the amount of disk space a file occupies, you can compress it with the bzip2 utility. The compression works especially well on files that contain patterns, as do most text files, but reduces the size of almost all files. The inverse of bzip2bunzip2restores a file to its original, decompressed form. Table 3-3 lists utilities that compress and decompress files. The bzip2 utility is the most efficient of these.

Table 3-3. (De)Compression utilities

bunzip2

Returns a file compressed with bzip2 to its original size and format (page 55)

bzcat

Displays a file compressed with bzip2 (page 55)

bzip2

Compresses a file (page 54)

compress

Compresses a file (not as well as gzip) (page 55)

gunzip

Returns a gzipped or compressed file to its original size and format (page 55)

gzcat

Displays a compressed file (page 55)

gzip

Compresses a file (page 55)

An archive is a file, frequently compressed, that contains a group of files. The tar utility (Table 3-4) packs and unpacks archives. The filename extensions .tar.bz2, .tar.gz, and .tgz identify compressed tar archive files and are often seen on software packages obtained over the Internet.

Table 3-4. Archive utility

tar

Creates or extracts files from an archive file (page 56)

The utilities listed in Table 3-5 determine the location of a utility on your system. For example, they can display the pathname of a utility or a list of C++ compilers available on your system.

Table 3-5. Location utilities

apropos

Searches the man page one-line descriptions for a keyword (page 59)

type

Locates a utility (page 58)

which

Displays the full pathname of a command you can run (page 58)

Table 3-6 lists utilities that display information about other users. You can easily learn a user's full name, the user's login status, the login shell of the user, and other items of information maintained by the system.

Table 3-6. User and system information utilities

finger

Displays detailed information about users who are logged in, including full names (page 61)

hostname

Displays your system name (page 43)

w

Displays detailed information about users who are logged in (page 62)

who

Displays information about users who are logged in (page 60)

The utilities shown in Table 3-7 can help you stay in touch with other users on the local network.

Table 3-7. User communication utilities

mesg

Permits or denies messages sent by write or talk (page 64)

write

Sends a message to another user who is logged in (page 63)

Table 3-8 lists miscellaneous utilities.

Table 3-8. Miscellaneous utilities

date

Displays the current date and time (page 52)

echo

Copies its arguments (page 921) to the terminal (page 52)


Exercises

1.

Which commands can you use to determine who is logged in on a specific terminal?

2.

How can you keep other users from using write to communicate with you? Why would you want to?

3.

What happens when you give the following commands if the file named done already exists?

$ cp to_do done
$ mv to_do done

4.

How can you find out which utilities are available on your system for editing files? Which utilities are available for editing on your system?

5.

How can you find the phone number for Ace Electronics in a file named phone that contains a list of names and phone numbers? Which command can you use to display the entire file in alphabetical order? How can you remove adjacent duplicate lines from the file? How can you remove all duplicates?

6.

What happens when you use diff to compare two binary files that are not identical? (You can use gzip to create the binary files.) Explain why the diff output for binary files is different from the diff output for ASCII files.

7.

Create a .plan file in your home directory. Does finger on your system display the contents of your .plan file?

8.

What is the result of giving the which utility the name of a command that resides in a directory that is not in your search path?

9.

Are any of the utilities discussed in this chapter located in more than one directory on your system? If so, which ones?

10.

Experiment by calling the file utility with the names of files in /usr/bin. How many different types of files are there?

11.

Which command can you use to look at the first few lines of a file named status.report? Which command can you use to look at the end of the file?


Advanced Exercises

12.

Re-create the colors.1 and colors.2 files used in Figure 3-8 on page 50. Test your files by running diff -u on them. Do you get the same results as in the figure?

13.

Try giving these two commands:

$ echo cat
$ cat echo

Explain the differences between them.

14.

Repeat exercise 5 using the file phone.gz, a compressed version of the list of names and phone numbers. Consider more than one approach to each question, and explain how you made your choices.

15.

Find existing files or create files that

  1. gzip compresses by more than 80 percent.

  2. gzip compresses by less than 10 percent.

  3. get larger when compressed with gzip.

Use ls -l to determine the sizes of the files in question. Can you characterize the files in a, b, and c?

16.

Older mail programs were not able to handle binary files. Suppose that you are mailing a file that has been compressed with gzip, which produces a binary file, and the recipient is using an old mail program. Refer to the man page on uuencode, which converts a binary file to ASCII. Learn about the utility and how to use it.

  1. Convert a compressed file to ASCII, using uuencode. Is the encoded file larger or smaller than the compressed file? Explain.

  2. Would it ever make sense to use uuencode on a file before compressing it? Explain.


Chapter 4. The Mac OS X Filesystem

Previous Page
Next Page

4. The Mac OS X Filesystem

IN THIS CHAPTER

The Hierarchical Filesystem | 72
Directory Files and Ordinary Files | 73
The Working Directory | 77
Your Home Directory | 77
Absolute Pathnames | 78
Relative Pathnames | 79
Directory Commands | 80
Access Permissions | 87
Extended Attributes | 93
Access Control Lists | 97
Types of Files | 99
Hard Links | 103
Symbolic Links | 105

A filesystem is a data structure (page 929) that usually resides on part of a disk and that holds directories of files. Filesystems store user and system data that are the basis of users' work on the system and the system's existence. This chapter discusses the organization and terminology of the Mac OS X filesystem, defines ordinary and directory files, and explains the rules for naming them. It also shows how to create and delete directories, move through the filesystem, and use absolute and relative pathnames to access files in various directories. It explores traditional UNIX file permissions, which allow you to share selected files with other users, as well as Mac OS X extended attributes: file forks, file attributes, and Access Control Lists (ACLs). It concludes with a discussion of different types of files and hard and symbolic links, which can make a single file appear in more than one directory.

Caution: Mac OS X files have multiple forks and attributes

Traditional UNIX systems have a simple mapping of a single filename to a single file. Macintosh operating systems divide files in a different way, assigning two or more separate files, called forks, to a single filename. The data fork is the traditional UNIX file and holds the file data. The resource fork holds a database of file resources (page 93).

Some UNIX utilities read the data fork only. Most utility programs provided with Mac OS X version 10.4 and later work properly with files that have multiple forks. Some third-party utilities may not handle resource forks correctly.

Similarly Mac OS X files have additional attributes, including type and creator codes, that some utilities do not recognize. See "File Attributes" on page 95 for more information.


The Hierarchical Filesystem

Family tree

A hierarchical (page 935) structure frequently takes the shape of a pyramid. One example of this type of structure is found by tracing a family's lineage: A couple has a child, who may in turn have several children, each of whom may have more children. This hierarchical structure is called a family tree (Figure 4-1).

Figure 4-1. A family tree

Directory tree

Like the family tree it resembles, the Mac OS X filesystem is called a tree. It consists of a set of connected files. This structure allows you to organize files so you can easily find any particular one. On a standard Mac OS X system, each user starts with one directory, to which the user can add subdirectories to any desired level. By creating multiple levels of subdirectories, a user can expand the structure as needed.

Subdirectories

Typically each subdirectory is dedicated to a single subject, such as a person, project, or event. The subject dictates whether a subdirectory should be subdivided further. For instance, Figure 4-2 shows a secretary's subdirectory named correspond. This directory contains three subdirectories: business, memos, and personal. The business directory contains files that store each letter the secretary types. If you expect many letters to go to one client, as is the case with milk_co, you can dedicate a subdirectory to that client.

Figure 4-2. A secretary's directories

One strength of the Mac OS X filesystem is its ability to adapt to users' needs. You can take advantage of this strength by strategically organizing your files so they are most convenient and useful for you.


Directory Files and Ordinary Files

Like a family tree, the tree representing the filesystem is usually pictured upside down, with its root at the top. Figures 4-2 and 4-3 show that the tree "grows" downward from the root, with paths connecting the root to each of the other files. At the end of each path is either an ordinary file or a directory file. Ordinary files, or simply files, appear at the ends of paths that cannot support other paths. Directory files, also referred to as directories or folders, are the points that other paths can branch off from. (Figures 4-2 and 4-3 show some empty directories.) When you refer to the tree, up is toward the root and down is away from the root. Directories directly connected by a path are called parents (closer to the root) and children (farther from the root). A pathname is a series of names that trace a path along branches from one file to another. More information about pathnames appears on page 78.

Figure 4-3. Directories and ordinary files

Filenames

Every file has a filename. The maximum length of a filename varies with the type of filesystem; Mac OS X supports several types of filesystems. Although most of today's filesystems allow you to create files with names up to 255 characters long, some filesystems restrict you to shorter names. While you can use almost any character in a filename, you will avoid confusion if you choose characters from the following list:

  • Uppercase letters (AZ)

  • Lowercase letters (az)

  • Numbers (09)

  • Underscore (_)

  • Period (.)

  • Comma (,)

/ or root

The root directory is always named / (slash) and referred to by this single character. No other file can use this name or have a / in its name. However, in a pathname, which is a string of filenames including directory names, the slash separates filenames (page 78).

Caution: Copying files from HFS+ and HFSX filesystems can garble filenames

The Mac OS HFS+ and HFSX filesystems (page 86) support filenames using the Unicode (page 959) character set, in the UTF-8 (page 960) encoding. Filenames that use extended characters from Unicode may become garbled when you copy the files to other types of filesystems.

Like the children of one parent, no two files in the same directory can have the same name. (Parents give their children different names because it makes good sense, but Mac OS X requires it.) Files in different directories, like the children of different parents, can have the same name.

The filenames you choose should mean something. Too often a directory is filled with important files with such unhelpful names as hold1, wombat, and junk, not to mention foo and foobar. Such names are poor choices because they do not help you recall what you stored in a file. The following filenames conform to the suggested syntax and convey information about the contents of the file:

  • correspond

  • january

  • davis

  • reports

  • 2001

  • acct_payable

Filename length

When you share your files with users on other systems, you may need to make long filenames differ within the first few characters. Systems running DOS or older versions of Windows have an 8-character filename body length and a 3-character filename extension length limit. Some UNIX systems have a 14-character limit and older Macintosh systems have a 31-character limit. If you keep the filenames short, they are easy to type; later you can add extensions to them without exceeding the shorter limits imposed by some filesystems. The disadvantage of short filenames is that they are typically less descriptive than long filenames.

Long filenames enable you to assign descriptive names to files. To help you select among files without typing entire filenames, shells support filename completion. For more information about this feature, see the "Filename Completion" tip on page 43.

Case sensitivity

You can use uppercase and/or lowercase letters within filenames, but be careful: Some filesystems are case sensitive. For example, the UFS filesystem is case sensitive, so files named JANUARY, January, and january represent three distinct files. The FAT filesystem (used mostly for removable media) is not case sensitive, so those three filenames represent the same file. The HFS+ filesystem, which is the default OS X filesystem, is case preserving but not case sensitive. Case preserving means that the filesystem remembers which capitalization you used when you created a file and displays the filename with that capitalization, but it accepts any capitalization to refer to the file. Under HFS+, the three example filenames refer to the same file. The HFS+ Extended (HFSX) filesystem is case sensitive and is fully supported only under Mac OS X Server. Under HFSX, the three example filenames refer to three different files.

Caution: Do not use SPACEs within filenames

Although you can use SPACEs within filenames, it is a poor idea to do so when you are working on the command line. Because a SPACE is a special character, you must quote it on a command line. Quoting a character on a command line can be difficult for a novice user and cumbersome for an experienced user. Furthermore, many shell scripts do not properly handle filenames with SPACEs in them. Use periods or underscores instead of SPACEs: joe.02.04.26, new_stuff.

Filename Extensions

A filename extension is the part of the filename following an embedded period. In the filenames listed in Table 4-1, filename extensions help describe the contents of the file. Some programs, such as the C programming language compiler, depend on specific filename extensions; in most cases, however, filename extensions are optional. Some applications use filename extensions to identify files, but many use type codes and creator codes (page 96). Use extensions freely to make filenames easy to understand. If you like, you can use several periods within the same filenamefor example, notes.4.10.01 or files.tar.gz.

Table 4-1. Filename extensions

Filename

Meaning of extension

compute

An executable file

compute.app

An executable file or a directory that contains an application bundle (an application program and its support files)

compute.c

A C programming language source file

compute.dmg

A disk image file that can be mounted the way you would mount a CD-ROM

compute.o

An object code file

compute.pkg

A package file for use with the Apple Installer application

memo.0410.txt

A text file

memo.bz2

A file compressed with bzip2 (page 54); view with bzcat or decompress with bunzip2

memo.gz

A file compressed with gzip (page 55); view with zcat or decompress with gunzip

memo.html

A file meant to be viewed using a Web browser, such as Safari

memo.pdf

A PDF file; display with the Preview application or use open (page 42)

memo.ps

A PostScript file; display with the Preview application or use open (page 42)

memo.tar.Z

A tar (page 56) archive of files compressed with compress (page 55)

memo.tbz2 or memo.tar.bz2

A tar archive of files compressed with bzip2 (page 54)

memo.tgz or memo.tar.gz

A tar archive of files compressed with gzip (page 55)

memo.Z

A file compressed with compress; use uncompress (or gunzip, page 55) to decompress

photo.gif, photo.jpg, photo.jpeg, photo.bmp, photo.tif, or photo.tiff

A file containing graphical information, such as a picture

Invisible Filenames

A filename that begins with a period is called an invisible filename (or an invisible file or sometimes a hidden file) because ls and the Mac OS X GUI do not normally display it. The command ls a displays all filenames, even invisible ones. Names of startup files (page 77) usually begin with a period so that they are invisible and do not clutter a directory listing. The .plan file (page 61) is also invisible. Two special invisible entriesa single and double period (. and ..)appear in every directory (page 82).

You can also hide files from the OS X GUI by setting the file's invisible flag (page 95) using SetFile (page 834). The OS X GUI does not normally display files that have this flag set, although ls does. Under Mac OS X 10.3 and earlier, files listed in the file named .hidden in the root directory are not displayed by the GUI. Files named .hidden in other directories are ignored. The .hidden file is not used under Mac OS 10.4 and later.

The Working Directory

pwd

While you are logged in on a character-based interface to a Mac OS X system, you are always associated with a directory. The directory you are associated with or are working in is called the working directory or current directory. Sometimes this association is referred to in a physical sense: "You are in (or working in) the max directory." The pwd (print working directory) utility displays the pathname of the working directory.

Your Home Directory

When you first log in on a Mac OS X system or start a Terminal window, your working directory is your home directory. To display the pathname of your home directory, use pwd just after you log in (Figure 4-4).

Figure 4-4. Logging in and displaying the pathname of your home directory
login: zach
Password:
Last login: Wed Oct 20 11:14:21 from bravo
Welcome to Darwin!
$ pwd
/Users/zach

When used without any arguments, the ls utility displays a list of the files in the working directory. Because your home directory is the only working directory you have used so far, ls has always displayed a list of files in your home directory. (All the files you have created up to this point were created in your home directory.)

Startup Files

Also called configuration files or preference files

Startup files, which appear in your home directory, give the shell and other programs information about you and your preferences. Mac OS and application documentation refer to startup files as configuration files or preference files. Frequently one of these files tells the shell what kind of terminal you are using (page 913) and executes the stty (set terminal) utility to establish the erase (page 25) and line kill (page 25) keys.

Either you or the system administrator can put a shell startup file containing shell commands in your home directory. The shell executes the commands in this file each time you log in. Because startup files have invisible filenames, you must use the ls a command to see whether one is present in your home directory. See pages 259 and 340 for more information about startup files.

Many Mac OS X applications store startup files in the Library and Library/Preferences subdirectories of your home directory, which are created when an account is set up. Most of these files do not have invisible filenames.


Pathnames

This section discusses absolute and relative pathnames and explains how to use them to your advantage.

Absolute Pathnames

Every file has a pathname. Figure 4-5 shows the pathnames of directories and ordinary files in part of a filesystem hierarchy. An absolute pathname always starts with a slash (/), the name of the root directory. You can build the absolute pathname of a file by tracing a path from the root directory through all the intermediate directories to the file. String all the filenames in the path together, separating each from the next with a slash (/) and preceding the entire group of filenames with a slash (/). This path of filenames is called an absolute pathname because it locates a file absolutely by tracing a path from the root directory to the file. The part of a pathname following the final slash is called a simple filename, or just filename.

Figure 4-5. Absolute pathnames

~ (Tilde) in Pathnames

In another form of absolute pathname, the shell expands the characters ~/ (a tilde followed by a slash) at the start of a pathname into the pathname of your home directory. Using this shortcut, you can display your .bashrc startup file (page 259) with the following command, no matter which directory is your working directory:

$ less ~/.bashrc

A tilde quickly references paths that start with your or someone else's home directory. The shell expands a tilde followed by a login name at the beginning of a pathname into the pathname of that user's home directory. For example, assuming he has permission to do so, Zach can examine Max's .bashrc file with the following command:

$ less ~max/.bashrc

Refer to "Tilde expansion" on page 324 for a more extensive discussion of this topic.

Carbon Pathnames

The Carbon programming environment was used on Macintosh systems prior to the introduction of Mac OS X. Carbon pathnames are radically different from OS X pathnames and are discussed here because many applications still use them.

The Carbon environment uses the historical Macintosh model of a desktop holding mounted disks, which does not correspond well to the UNIX filesystem's single directory hierarchy. Furthermore, the Carbon environment represents pathnames differently from the UNIX environment. Carbon pathnames use a colon (:) as a separator, instead of a slash (/). No top-level root exists for Carbon pathnames; instead, they start with the name of a disk. For example, the UNIX-style pathname /Volumes/Images/backups equates to the Carbon-style pathname Images:backups.

Caution: Know which kind of pathname you are using

If you enter a UNIX pathname when a Carbon pathname is expected, or vice versa, the system may display an error message, not retrieve the correct file, or save the file in the wrong location.

Relative Pathnames

A relative pathname traces a path from the working directory to a file. The pathname is relative to the working directory. Any pathname that does not begin with the root directory (/) or a tilde (~) is a relative pathname. Like absolute pathnames, relative pathnames can describe a path through many directories. The simplest relative pathname is a simple filename. A simple filename identifies a file in the working directory. The examples in the next sections use absolute and relative pathnames.

Caution: When using a relative pathname, know which directory is the working directory

The location of the file that you are accessing with a relative pathname is dependent on (is relative to) the working directory. Make sure you know which directory is the working directory before you use a relative pathname. You can use pwd to verify the directory. If you are using mkdir and you are not where you think you are in the file hierarchy, the new directory will end up in an unexpected location.

It does not matter which directory is the working directory when you use an absolute pathname.

Significance of the Working Directory

To access any file in the working directory, you need only a simple filename. To access a file in another directory, however, you must use a pathname. Typing a long pathname is tedious and increases the chance of making a mistake. This possibility is less likely when you are using a GUI, where you click filenames or icons. You can choose a working directory for any particular task to reduce the need for long pathnames. Your choice of a working directory does not allow you to do anything you could not do otherwise, but it does make some operations easier.

Refer to Figure 4-6 as you read this paragraph. Files that are children of the working directory can be referenced by simple filenames. Grandchildren of the working directory can be referenced by short relative pathnames: two filenames separated by a slash. When you manipulate files in a large directory structure, short relative pathnames can save time and aggravation. If you choose a working directory that contains the files used most often for a particular task, you need to use fewer long, cumbersome pathnames.

Figure 4-6. Relative pathnames

Previous Page
Next Page


Directory Commands

This section discusses how to create directories (mkdir), switch between directories (cd), remove directories (rmdir), use pathnames to make your work easier, and move and copy files and directories between directories.

mkdir: Creates a Directory

The mkdir utility creates a directory. The argument (page 921) to mkdir becomes the pathname of the new directory. The following examples develop the directory structure shown in Figure 4-7. In the figure, the directories that are added appear in a lighter shade than the others and are connected by dashes.

Figure 4-7. The file structure developed in the examples

In Figure 4-8, pwd shows that Zach is working in his home directory (/Users/zach) and ls shows the names of the files in his home directory: demo, names, and temp. Using mkdir, Zach creates a directory named literature as a child of his home directory. He uses a relative pathname (a simple filename) because he wants the literature directory to be a child of the working directory. Of course, Zach could have used an absolute pathname to create the same directory: mkdir/Users/zach/literature.

Figure 4-8. The mkdir utility
$ pwd
/Users/zach
$ ls
demo  names  temp
$ mkdir literature
$ ls -F
demo  literature/  names  temp
$ ls literature
$

The second ls in Figure 4-8 verifies the presence of the new directory. The F option to ls displays a slash after the name of each directory and an asterisk after each executable file (shell script, utility, or application). When you call it with an argument that is the name of a directory, ls lists the contents of the directory. The final ls does not display anything because there are no files in the literature directory.

The following commands show two ways to create the promo directory as a child of the newly created literature directory. The first way checks that /Users/zach is the working directory and uses a relative pathname:

$ pwd
/Users/zach
$ mkdir literature/promo

The second way uses an absolute pathname:

$ mkdir /Users/zach/literature/promo

Use the p (parents) option to mkdir to create both the literature and promo directories with one command:

$ pwd
/Users/zach
$ ls
demo  names  temp
$ mkdir p literature/promo

or

$ mkdir p /Users/zach/literature/promo

cd: Changes to Another Working Directory

The cd (change directory) utility makes another directory the working directory but does not change the contents of the working directory. Figure 4-9 shows two ways to make the /Users/zach/literature directory the working directory, as verified by pwd.

Figure 4-9. cd changes your working directory
$ cd /Users/zach/literature
$ pwd
/Users/zach/literature
$ cd
$ pwd
/Users/zach
$ cd literature
$ pwd
/Users/zach/literature

First Zach uses cd with an absolute pathname to make literature his working directory. It does not matter which is your working directory when you give a command with an absolute pathname. A pwd command confirms the change. When used without an argument, cd makes your home directory the working directory, as it was when you first logged in. The second cd command in Figure 4-9 does not have an argument and makes Zach's home directory the working directory. Finally, knowing that he is working in his home directory, Zach uses a simple filename to make the literature directory his working directory (cd literature) and confirms the change with pwd.

The . and .. Directory Entries

The mkdir utility automatically puts two entries in each directory you create: a single period (.) and a double period (..). The . is synonymous with the pathname of the working directory and can be used in its place; the .. is synonymous with the pathname of the parent of the working directory. These entries are invisible because each of their filenames begins with a period.

Tip: The working directory versus your home directory

The working directory is not the same as your home directory. Your home directory remains the same for the duration of your session and usually from session to session. Each time immediately after you log in, you are working in the same directory: your home directory.

Unlike your home directory, your working directory can change as often as you like. You have no set working directory, which explains why some people refer to it as the current directory. When you log in and until you change directories by using cd, your home directory is your working directory. If you were to change directories to Max's home directory, then Max's home directory would be your working directory.

With the literature directory as the working directory, the following example uses .. three times: first to list the contents of the parent directory (/Users/zach), second to copy the memoA file to the parent directory, and third to list the contents of the parent directory again.

$ pwd
/Users/zach/literature
$ ls
memoA  promo
$ ls ..
demo  literature  names  temp
$ cp memoA ..
$ ls ..
demo  literature  memoA  names  temp

After using cd to make promo his working directory, Zach can use a relative pathname to call vim (page 147) to edit a file in his home directory:

$ cd promo
$ vim ../../names

You can use an absolute or relative pathname or a simple filename virtually anywhere that a utility or program requires a filename or pathname. This usage holds true for ls, vim, mkdir, rm, and most other Mac OS X utilities.

rmdir: Deletes a Directory

The rmdir (remove directory) utility deletes a directory. You cannot delete the working directory or a directory that contains files other than the . and .. entries. If you need to delete a directory that has files in it, first use rm to delete the files and then delete the directory. You do not have to (nor can you) delete the . and .. entries; rmdir removes them automatically. The following command deletes the promo directory:

$ rmdir /Users/zach/literature/promo

The rm utility has a r option (rm r filename) that recursively deletes files, including directories, within a directory and also deletes the directory itself.

Caution: Use rm r carefully, if at all

Although rm r is a handy command, you must use it carefully. Do not use it with an ambiguous file reference such as *. It is frighteningly easy to wipe out your entire home directory with a single short command.

Using Pathnames

touch

Use a text editor to create a file named letter if you want to experiment with the examples that follow. Alternatively, you can use touch (page 877) to create an empty file:

$ cd
$ pwd
/Users/zach
$ touch letter

With /Users/zach as the working directory, the following example uses cp with a relative pathname to copy the file letter to the /Users/zach/literature/promo directory (you will need to create promo again if you deleted it earlier). The copy of the file has the simple filename letter.0610:

$ cp letter literature/promo/letter.0610

If Zach does not change to another directory, he can use vim to edit the copy of the file he just made:

$ vim literature/promo/letter.0610

If Zach does not want to use a long pathname to specify the file, he can use cd to make promo the working directory before using vim:

$ cd literature/promo
$ pwd
/Users/zach/literature/promo
$ vim letter.0610

To make the parent of the working directory the new working directory, Zach can give the following command, which takes advantage of the .. directory entry:

$ cd ..
$ pwd
/Users/zach/literature

mv, cp: Moves or Copies a File

Chapter 3 discussed the use of mv to rename files. However, mv works more generally than that: You can use this utility to move files from one directory to another (change the pathname of a file) as well as to change a simple filename. When used to move one or more files to a new directory, the mv command has this syntax:

mv existing-file-list directory

If the working directory is /Users/zach, Zach can use the following command to move the files names and temp from the working directory to the literature directory:

$ mv names temp literature

This command changes the absolute pathnames of the names and temp files from /Users/zach/names and /Users/zach/temp to /Users/zach/literature/names and /Users/zach/literature/temp, respectively (Figure 4-10). Like most OS X commands, mv accepts either absolute or relative pathnames.

Figure 4-10. Using mv to move names and temp

As you work with Mac OS X and create more files, you will need to create directories using mkdir to keep those files organized. The mv utility is a useful tool for moving files from one directory to another as you develop your directory hierarchy.

The cp utility works the same way that mv does, except that it makes copies of the existing-file-list in the specified directory.

mv: Moves a Directory

Just as it moves ordinary files from one directory to another, so mv can move directories. The syntax is similar except that you specify one or more directories, not ordinary files, to move:

mv existing-directory-list new-directory

If new-directory does not exist, the existing-directory-list must contain just one directory name, which mv changes to new-directory (mv renames the directory). Although directories can be renamed using mv, their contents cannot be copied with cp unless you use the r option. Refer to the sections on ditto (page 44), cpio (page 693), pax (page 809), and tar (page 862) for other ways to copy and move directories.


Filesystems

Mac OS X supports several types of filesystems. The most common is the default HFS+ (Hierarchical File System plus). Introduced under Mac OS 8.1 to support larger disks, HFS+ is an enhanced version of the original HFS filesystem. When it was introduced in OS 3.0 in 1986, HFS stood in contrast to the then-standard MFS (Macintosh File System). Some applications will not run correctly under filesystems other than HFS+.

OS X also supports traditional UNIX filesystems such as UFS (UNIX File System), which it inherited from Berkeley UNIX. HFS+ is different from traditional UNIX filesystems, but because UNIX offers a standardized filesystem interface, the differences are generally transparent to the user. The most significant difference lies in the handling of extended attributes (page 93).

Other supported filesystems include FAT16 and FAT32, which were originally used in MS-DOS and Windows. These filesystems are typically used for removable media, such as digital camera storage cards. ISO 9660 is used for CD-ROMs and the related UDF filesystem is often used for DVDs.

Nondisk Filesystems

Mac OS X supports filesystems that do not correspond to a physical volume such as a partition on a hard disk or a CD-ROM. A .dmg (disk image) file is one example (you can mount a disk image file so that you can access the files it holds by double-clicking it in the Finder). Another example is a virtual filesystem in which the filenames represent kernel functionality. For instance, the /Network virtual filesystem holds a directory tree representing the active local AppleTalk network. Although you can generally interact with a nondisk filesystem the same way you interact with any other filesystem, there may be some limitations. For example, you may not be able to write to a mounted .dmg file.

/Volumes

Each physical hard disk in a Mac OS X system is typically divided into one or more logical sections (partitions) that, when formatted, are called volumes. On a traditional UNIX system they are called filesystems.

Startup disk

Each OS X system has a volume, called the startup disk, that the system boots from. By default the startup disk is named Macintosh HD. When the system boots, Macintosh HD is mounted as the root directory (/).

The root directory always has a subdirectory named Volumes. For historical reasons, every volume other than the startup disk is mounted in the /Volumes directory. The pathname of a disk labeled Images is /Volumes/Images.

To simplify access, /Volumes holds a symbolic link (page 105) to the startup disk (the root directory). Assuming that the startup disk is named Macintosh HD, /Volumes/Macintosh HD is a symbolic link to /.

$ ls -ld '/Volumes/Macintosh HD'
lrwxr-xr-x   1 root  admin  1 Jul 12 19:03 /Volumes/Macintosh HD -> /

The /Volumes directory holds the same files as the Finder's top level so that the /Volumes directory serves as the effective root for the Finder and other applications. The Finder presents the pre-UNIX Mac OS view of the filesystem. The files in /Volumes are the same as the first components of Carbon pathnames (page 79). The system automatically mounts all volumes in /Volumes.


Access Permissions

Mac OS X supports two methods of controlling who can access a file and how they can access it: traditional UNIX access permissions (called permissions) and Access Control Lists (ACLs, page 97). ACLs provide finer-grained control of access privileges. You can also set file flags to limit a file's use. This section describes access permissions and file flags.

Three types of users can access a file: the owner of the file (owner), a member of a group to which the owner belongs (group), and everyone else (other). A user can attempt to access an ordinary file in three ways: by trying to read from, write to, or execute it. Three types of users, each of whom is able to access a file in three ways, equals a total of nine possible ways to access an ordinary file.

ls -l: Displays Permissions

When you call ls with the l option and the name of an ordinary file, ls displays a line of information about the file. The following example displays information for two files. The file letter.0610 contains the text of a letter, and check_spell contains a shell script, a program written in a high-level shell programming language:

$ ls -l letter.0610 check_spell
-rw-r--r--  1 zach  zach  3355  May  2 10:52 letter.0610
-rwxr-xr-x  2 zach  zach   852  May  5 14:03 check_spell

From left to right, the lines that an ls l command displays contain the following information (refer to Figure 4-11):

  • The type of file (first character)

  • The file's access permissions (the next nine characters)

  • The ACL flag (present if the file has an ACL)

  • The number of links to the file (see page 102)

  • The name of the owner of the file (usually the person who created the file)

  • The name of the group that has group access to the file

  • The size of the file in characters (bytes)

  • The date and time the file was created or last modified

  • The name of the file

Figure 4-11. The columns displayed by an ls l command

The type of file (first column) for letter.0610 is a hyphen (-) because it is an ordinary file (directory files have a d in this column).

The next three characters specify the access permissions for the owner of the file: r indicates read permission and w indicates write permission. The in the next column indicates that the owner does not have execute permission; otherwise, an x would appear here.

In a similar manner the next three characters represent permissions for the group, and the final three characters represent permissions for other (everyone else). In the preceding example, the owner of letter.0610 can read from and write to the file, whereas the group and others can only read from the file and no one is allowed to execute it. Although execute permission can be allowed for any file, it does not make sense to assign execute permission to a file that contains a document, such as a letter. The check_spell file is an executable shell script, and execute permission is appropriate. (The owner, group, and others have execute access permission.)

An lsl command displays a plus sign (+) between the permissions and the number of links for files that have an ACL (page 97), even if the list is empty.

chmod: Changes Access Permissions

The owner of a file controls which users have permission to access the file and how they can access it. When you own a file, you can use the chmod (change mode) utility to change access permissions for that file. In the following example, chmod adds (+) read and write permissions (rw) for all (a) users:

$ chmod a+rw letter.0610
$ ls -l letter.0610
-rw-rw-rw-  1 zach  zach  3355  May  2 10:52 letter.0610

Tip: You must have read permission to execute a shell script

Because a shell needs to read a shell script (a text file containing shell commands) before it can execute the commands within the script, you must have read permission for the file containing the script to execute it. You also need execute permission to execute a shell script directly on the command line. Binary (program) files do not need to be read; they are executed directly. You need only execute permission to run a binary (nonshell) program.

In the next example, chmod removes (-) read and execute (rx) permissions for users other (o) than the owner of the file (Zach) and members of the group associated with the file (zach):

$ chmod o-rx check_spell
$ ls -l check_spell
-rwxr-x---  2 zach  zach  852  May  5 14:03 check_spell

In addition to a (all) and o (other), you can use g (group) and u (user, although user refers to the owner of the file who may or may not be the user of the file at any given time) in the argument to chmod. Refer to page 265 for more information on using chmod to make a file executable.

Tip: chmod: o for other, u for owner

When using chmod, many people assume that the o stands for owner; it does not. The o stands for other, whereas u stands for owner (user).

The Mac OS X file access permission scheme lets you give other users access to the files you want to share yet keep your private files confidential. You can allow other users to read from and write to a file (you may be one of several people working on a joint project). You can allow others only to read from a file (perhaps a project specification you are proposing). Or you can allow others only to write to a file (similar to an inbox or mailbox, where you want others to be able to send you mail but do not want them to read your mail). Similarly you can protect entire directories from being scanned (covered shortly).

There is an exception to the access permissions just described. Anyone who knows the root password or is in the admin group can log in as Superuser (page 427) or use sudo (page 431) and gain full access to all files, regardless of owner or access permissions.

Groups

Groups (page 934) allow users to share files or programs without allowing all system users access to them. This scheme is useful if several users are working with files that are not public. In a traditional UNIX system, the /etc/group file associates one or more usernames with each group (number). Under OS X this file is used in single-user mode only. When OS X is in multiuser mode it relies on NetInfo (page 441) to provide group information.

Each user has a primary group, the group that user is assigned in the passwd database. By default Mac OS X has user private groups: Each user's primary group has the same name as the user's username. For example, the user with the username max is a member of the group named max. In addition, a user can belong to other groups. In effect, you simultaneously belong to both your primary group and any groups you are assigned to. When you attempt to access a file that you do not own, the operating system checks whether you are a member of the group that has access to the file. If you are, your access permissions are controlled by the group access permissions for the file. If you are not a member of the group that has access to the file and you do not own the file, you are subject to the public (other) access permissions for the file.

When you create a new file, it is assigned to the group associated with the directory the file is being written into, even if you do not belong to that group. You can change the group that a file is associated with by using the chgrp command (page 675), but you can change it only to a group of which you are a member (unless you are working as root). You can see which groups a user is in by using the groups utility:

$ groups zach
zach admin

Setuid and Setgid Permissions

When you execute a file that has setuid (set user ID) permission, the process executing the file takes on the privileges of the file's owner. For example, if you run a setuid program that removes all files in a directory, you can remove files in any of the file owner's directories, even if you do not normally have permission to do so.

In a similar manner, setgid (set group ID) permission means that the process executing the file takes on the privileges of the group the file is associated with. The ls utility shows setuid permission by placing an s in the owner's executable position and setgid permission by placing an s in the group's executable position:

$ ls -l program1
-rwxr-xr-x  1 zach  zach  15828 Nov  5 06:28 program1
$ chmod u+s program1
$ ls -l program1
-rwsr-xr-x  1 zach  zach  15828 Nov  5 06:28 program1
$ chmod g+s program1
$ ls -l program1
-rwsr-sr-x  1 zach  zach  15828 Nov  5 06:28 program1

Directory Access Permissions

Access permissions have slightly different meanings when they are used with directories. Although the three types of users can read from or write to a directory, the directory cannot be executed. Execute access permission is redefined for a directory: It means that you can cd into the directory and/or examine files that you have permission to read from in the directory. It has nothing to do with executing a file.

Security: Minimize use of setuid and setgid programs owned by root

Executable files that are setuid and owned by root have Superuser privileges when they are run, even if they are not run by root. This type of program is very powerful because it can do anything that Superuser can do (that the program is designed to do). Similarly executable files that are setgid and belong to the group root have extensive privileges.

Because of the power they hold and their potential for destruction, avoid creating indiscriminately and using setuid and setgid programs owned by or belonging to the group root. Because of their inherent dangers, many sites do not allow these programs on their systems. See page 429 for information on setuid files owned by root.

Security: Do not write setuid shell scripts

Never give shell scripts setuid permission. Several techniques for subverting them are well known.

When you have only execute permission for a directory, you can use ls to list a file in the directory if you know its name. You cannot use ls without an argument to list the entire contents of the directory. In the following exchange, Max first verifies that he is logged in as himself. Then he checks the permissions on Zach's info directory. You can view the access permissions associated with a directory by running ls with the d (directory) and l (long) options:

$ who am i
max      ttyp1    Jul 31 22:16 (bravo.example.co)
$ ls -ld /Users/zach/info
drwx-----x   4 zach  zach  136 Aug  1 12:00 /Users/zach/info
$ ls -l /Users/zach/info
ls: info: Permission denied

The d at the left end of the line that ls displays indicates that /Users/zach/info is a directory. Zach has read, write, and execute permissions; members of the zach group have no access permissions; and other users have execute permission only, as indicated by the x at the right end of the permissions. Because Max does not have read permission for the directory, the ls l command returns an error.

When Max specifies the names of the files he wants information about, he is not reading new directory information but rather searching for specific information, which he is allowed to do with execute access to the directory. He has read access to notes so he has no problem using cat to display the file. He cannot display financial because he does not have read access to it:

$ ls -l /Users/zach/info/financial /Users/zach/info/notes
-rw-------   1 zach  zach  39 Aug  1 12:00 /Users/zach/info/financial
-rw-r--r--   1 zach  zach  30 Aug  1 12:00 /Users/zach/info/notes
$ cat /Users/zach/info/financial
cat: /Users/zach/info/financial: Permission denied
$ cat /Users/zach/info/notes
This is the file named notes.

Next Zach gives others read access to his info directory:

$ chmod o+r /Users/zach/info

When Max checks his access permissions on info, he finds that he has both read and execute access to the directory. Now ls l works without arguments, but he still cannot read financial. (This is an issue of file permissions, not directory permissions.) Finally Max tries to create a file named newfile by using touch (page 877). If Zach were to give him write permission to the info directory, Max would be able to create new files in it:

$ ls -ld /Users/zach/info
drwx---r-x   4 zach  zach  136 Aug  1 12:00 /Users/zach/info
$ ls -l /Users/zach/info
total 16
-rw-------  1 zach  zach  39 Aug  1 12:00 financial
-rw-r--r--  1 zach  zach  30 Aug  1 12:00 notes
$ cat /Users/zach/info/financial
cat: /Users/zach/info/financial: Permission denied
$ touch /Users/zach/info/newfile
touch: /Users/zach/info/newfile: Permission denied

File Flags

In addition to traditional UNIX permissions and ACLs, Mac OS X supports BSD file flags. File flags primarily limit a file's use. An ls lo command displays file flags between the group and size fields. The memo.txt file has the nodump and uchg file flags set:

$ ls -lo memo.txt
-rw-r--r--   1 sam  sam  uchg,nodump 82556 Aug  1 12:45 memo.txt

The uchg file flag prevents anyone from changing or removing a file, and nodump prevents a file from being dumped. The following example shows the contents of the sample file and shows that it has no file flags set (the hyphen in the file flags position indicates that no file flags are set):

$ cat sample
This is a sample file.
$ ls -lo sample
-rw-r--r--   1 sam  sam  - 23 Aug  1 12:35 sample

Tip: The uchg flag is the same as the locked attribute

Although BSD flags and attribute flags (page 95) are not generally the same, the uchg flag and the locked attribute are interchangeable. Setting one sets the other and clearing one clears the other.

chflags

Next chflags sets the uchg file flag on sample. Once the uchg file flag is set, no one can modify or remove the file. This file flag overrides the traditional UNIX permissions that would normally allow you to remove a file from a directory that you own:

$ chflags uchg sample
$ ls -lo sample
-rw-r--r--   1 sam  sam  uchg 23 Aug  1 12:35 sample
$ cp memo.txt sample
cp: sample: Operation not permitted
$ rm sample
override rw-r--r--  sam/sam uchg for sample? y
rm: sample: Operation not permitted

You clear a flag the same way that you set it, except you precede the name of the flag with the string no:

$ chflags nouchg sample
$ rm sample
$

File flags cannot be overridden by anyone, not even root. The uchg file flag is a user file flag. It can be set and cleared by the owner of a file or root. The schg file flag is a system file flag that protects a file the way the uchg user file flag does. System file flags can be set only by root and can be cleared by root only in single-user mode. For more information on file flags, see the chflags man page.


Extended Attributes

This section describes resource forks, file attributes, and ACLs, which are collectively known as extended attributes.

Resource forks and file attributes are native to the HFS+ filesystem. Mac OS X emulates resource forks and file attributes on other types of filesystems. These features are not found on traditional UNIX filesystems.

Caution: Some utilities do not process extended attributes

Some third-party programs and most utilities under Mac OS X 10.3 and earlier do not support extended attributes. Some utilities require options to process extended attributes.

Some utilities, on systems with ACLs enabled, have a bug involving extended attributes. For more information see the "Enabling ACLs may cause errors" caution on page 98.

See also the "Redirection does not support resource forks" tip on page 94.

File Forks

Forks are segments of a single file, each holding different content. Mac OS has supported file forks since its inception. The most widely used are the data fork and the resource fork.

Data forks

The data fork is equivalent to a traditional UNIX file. It consists of an unstructured stream of bytes. Many files have only a data fork.

Resource forks

A few files have only a resource fork. The resource fork holds a database that allows random access to resources, each of which has a type and identifier number. Modifying, adding, or deleting one resource has no effect on other resources. Resource forks can store different types of informationsome critical and some merely useful. For example, a Mac OS graphics program might save a smaller copy of an image (a preview or thumbnail) in a resource fork. Also, text files created with the BBEdit text editor store display size and tab stop information in the file's resource fork. Because this program is a text editor and not a word processor, this type of information cannot be stored in the data fork. Losing a resource fork that holds a thumbnail or display information is at most an inconvenience because the thumbnail can be regenerated from the original image. Other programs store more important data in the resource fork or create files that contain only a resource fork, which contains all of the file's information. In this case, losing the resource fork is just as bad as losing the data fork. It may even be worse: You may not notice when a resource fork is missing because the data fork is still there. You might notice the loss only when you try to use the file.

A traditional UNIX filesystem associates each filename with a single stream of bytes. Forks do not fit this model. As a result, many UNIX utilities do not process resource forks but process only data forks. Most of the file utilities provided with OS X 10.4 and later support resource forks. Many third-party utilities do not support resource forks.

Caution: Redirection does not support resource forks

When you redirect input to or output from a utility (page 122), only the information in the data fork is redirected. Information in the resource fork is not redirected. For example, the following command copies the data fork of song.ogg only:

$ cat song.ogg > song.bak.ogg

In a similar manner, pipes do not redirect information other than that in the data fork.

Optional

..namedfork

Under Mac OS X 10.4 and later you can verify the existence of file forks on an HFS+ filesystem. The ..namedfork pseudodirectory, as a subdirectory of a file, holds files that correspond to a file's forks. The two most common files are data, which holds the data fork, and rsrc, which holds the resource fork. The following example shows how to use ..namedfork to verify the existence of and display the size of the data and resource forks associated with the pic.jpg file. You cannot list the ..namedfork pseudodirectory:

$ ls -l pic.jpg
-rwxr-xr-x   1 max  max  13432 Jul 18 15:15 pic.jpg
$ ls -l pic.jpg/..namedfork
ls: pic.jpg/..namedfork: Not a directory
$ ls -l pic.jpg/..namedfork/data
-rwxr-xr-x   1 max  max  13432 Jul 18 15:15 pic.jpg/..namedfork/data
$ ls -l pic.jpg/..namedfork/rsrc
-rwxr-xr-x   1 max  max  140622 Jul 18 15:15 pic.jpg/..namedfork/rsrc

The data file is human readable if the underlying file is human readable. The rsrc file is not in a human-readable format.

If you are unsure whether a program supports resource forks, test it before relying on it. Make a backup copy using the Finder, ditto, or, under version 10.4 and later, cp. Then try the program you want to test; for example, use it to make a copy of your test file. You can then check whether the modified or copied file works correctly.

Table 4-2 lists utilities that manipulate resource forks. These utilities are installed with the Developer Tools package. Consult the respective man pages for detailed descriptions.

Table 4-2. Utilities that manipulate resource forks

Utility

Function

Rez

Creates a resource fork from a resource description file

DeRez

Creates a resource description file from a resource fork

RezWack

Converts a file with forks to a single flat file that holds all the forks

UnRezWack

Converts a flat file that holds forks to a file with forks

SplitForks

Converts a file with forks to multiple files, each holding a fork

File Attributes

A file contains data. Information about the file is called metadata. Examples of metadata are ownership information and permissions for a file. Mac OS X stores more metadata than other UNIX and UNIX-like systems store. This section discusses file attributes, the metadata stored by Mac OS X.

File attributes include the following items:

  • Attribute flags

  • Type codes

  • Creator codes

The same caveats apply to file attributes as apply to resource forks: Some utilities may not preserve them when copying or manipulating files, and many utilities do not recognize attribute flags. Loss of file attributes is particularly common when moving files to non-Macintosh systems.

Attribute Flags

Attribute flags (Table 4-3) hold information that is distinct from traditional UNIX permissions. Two attribute flags are the invisible flag (which keeps a file from being displayed in file dialogs and in the Finder) and the locked flag (which keeps a file from being modified). Flags are generally ignored by utilities and affect only applications. The ls utility lists files that have the invisible flag set. See GetFileInfo on page 749 and SetFile on page 834 for information on displaying, setting, and clearing attribute flags. Some of the flags in Table 4-3 are meaningful only under Mac OS 9 and earlier and have no effect under Mac OS X.

Table 4-3. Attribute flags

Flag

Can the flag be set on a folder?

Description

a

no

Alias file

b

no

Has bundle

c

yes

Custom icon

d

yes

File located on desktop

e

yes

Extension is hidden

i

yes

Inited (Finder has placed file in a window)

l

no

Locked

m

no

Shared (can be run more than one time)

n

no

File has no INIT resource

s

no

System file

t

no

Stationery pad file

v

yes

Invisible

z

yes

Busy

Type Codes and Creator Codes

Type codes and creator codes are 32-bit integers, generally displayed as 4-character names, that specify the type and creator of a file. The creator code specifies the application that was used to create a document, not the user who created it. An application can typically open documents that have the same creator code as the application, but not documents with other creator codes.

Creator codes generally correspond to vendors or product lines. The operating systemand in particular the Finderuses creator codes to group related files. For example, the AppleWorks application file and its document files have the creator code BOBO. In the file browser in an application's open file dialog box, grayed-out files generally indicate files that have a different creator code from the application. Finally, the open utility (page 803) looks at creator codes when it opens a file.

Type codes indicate how a file is used. The type code APPL indicates an application, a program used to open other files. An AppleWorks word processor document has the type code CWWP, a mnemonic for Claris Works Word Processor (AppleWorks used to be named Claris Works). While a few type codes, such as the application type, are standardized, vendors are free to invent new type codes for their programs to use. A single application may support several document types. For example, AppleWorks supports spreadsheet files (CWSS), word processor documents (CWWP), and drawings (CWGR). Similarly a graphics program will typically support many document types. Data files used by an application may also have the same creator code as the application, even though they cannot be opened as documents. For example, the dictionary used by a spell checker cannot be opened as a document but typically uses the same creator code as the spell checker.

Filename extensions

Filename extensions (page 75) can substitute for type and creator codes. For example, an AppleWorks word processor document is saved with an extension of .cwk. Also, if open cannot determine which application to use to open a file using the file's creator code, it reverts to using the file's filename extension.

Access Control Lists

Access Control Lists (ACLs), introduced under Mac OS X 10.4, augment standard UNIX file permissions by allowing you to control file access on a per-user or per-group basis. An ACL is an extended attribute (page 93) that holds access rules, each of which grants or denies file access to a user or group. You can give as many users or groups read, write, and/or execute access as you desire. ACLs may not work on a network (remote) filesystem depending on whether the remote operating system supports OS X ACLs.

One use for ACLs is to control access to system files. For example, you could give Max and Zach read and write access to the message of the day file (/etc/motd) so that either of them could maintain it. You would not have to set up a special group or give write access to the public, as you would on a traditional UNIX system.

Security: Grant permissions to root-owned files selectively

Grant write and execute access to files owned by root with care. You are giving a user the ability to do things that root normally does. Keep a log of your ACL permission assignments.

Enabling and Disabling ACLs

Under Mac OS X, ACL support is not enabled by default. You must use fsaclctl (file system ACL control) to enable ACL support for each volume (filesystem) that you want to use ACLs on. The fsaclctl utility operates on a single volume using the p path option or on all mounted volumes using the a option. With no other options, fsaclctl displays the state of ACL support on a volume. You must be working as root or use sudo to enable and disable ACLs. The e option to fsaclctl enables ACLs on the specified volume(s), and the d option disables them:

# fsaclctl -p /
Access control lists are not supported or currently disabled on /.
# fsaclctl -p / -e
# fsaclctl -p /
Access control lists are supported on /.

Caution: Enabling ACLs may cause errors

Enabling ACLs on a filesystem prevents some programs that handle extended attributes (page 93) from working correctly. If a file has a resource fork but no ACL, attempts to copy it may fail with an error message such as the following:

$ pax -w -f arch.pax picture.jpg
pax: Unable to open ._picture.jpg to read <No such file or directory>

Some utilities, such as cp, are not affected by this problem.

How ACLs Work

Each access rule in an ACL specifies a user or group, the word allow or deny, and one or more actions such as read, write, and execute. When a user attempts to access a file, OS X tests in turn each rule in that file's ACL. If it finds a match for the user attempting to access the file and for the action that the user is attempting to perform on the file, the allow or deny word in that rule determines whether the user can access the file. If no match is found, the next rule is tested. If no rules match, OS X uses the traditional UNIX file permissions. Because ACL rules are applied before traditional UNIX permissions, ACL rules override traditional UNIX permissions.

Caution: cp does not copy ACLs

Regardless of which options you specify, cp does not copy ACLs. The Finder does not copy ACLs either. However, the tar utility (page 862) copies and restores ACLs.

Creating ACL Rules

In the following example, Sam uses ls l to display permissions for the report file. You can tell that this file has no ACL because no plus sign (+) appears to the right of the permissions:

$ ls l report
-rw-r--r--   1 sam  sam  0 Jun 26 14:52 report

chmod

An extension to chmod allows you to add, remove, and modify ACL rules for a file. Sam uses chmod to add (+a) an access control rule so that Zach can read from and write to report. Refer to "ACLs" on page 678 for more information on using chmod to control ACLs. The output of ls l now shows a plus sign following the permissions, indicating that the file has an ACL. The e option to ls, when used with l, displays a file's ACL.

$ chmod +a "zach allow read,write" report
$ ls -l report
-rw-r--r-- + 1 sam  sam  0 Jun 26 14:52 report
$ ls -le report
-rw-r--r-- + 1 sam  sam  0 Jun 26 14:52 report
 0: user:zach allow read,write

Mac OS X controls access to files based on ACLs. As with other extended attributes, some utilities and programs may not copy or preserve ACLs when working with a file. See page 678 for more information on ACLs.


Types of Files

Mac OS X supports many types of files. This section discuss the following types of files:

  • Ordinary files, directories, links, and inodes (page 99)

  • Symbolic links (page 99)

  • Special files (page 100)

  • FIFO special files (named pipes) (page 101)

  • Sockets (page 101)

  • Block and character devices (page 101)

This section does not discuss file type codes that identify the contents of ordinary files. Type codes are discussed on page 96.

Ordinary Files, Directories, Links, and Inodes

Ordinary and directory files

An ordinary file stores user data, such as textual information, programs, or an image, such as a jpeg or tiff file. A directory is a standard-format disk file that stores information, including names, about ordinary files and other directory files.

Inodes

An inode is a data structure (page 929), stored on disk, that defines a file's existence and is identified by an inode number. A directory relates each of the filenames it stores to a specific inode. An inode contains critical information, such as the name of the owner of the file, the file's physical location on the disk, and the number of hard links that point to it. HFS+ filesystems do not use inodes but emulate their behavior.

When you move (mv) a file within a filesystem, you change the filename portion of the directory entry associated with the inode that describes the file. You do not create a new inode. If you move a file to another filesystem, mv first creates a new inode on the destination filesystem and then deletes the original inode. You can also use mv to move a directory recursively, in which case all the files are copied and deleted.

Hard links

When you make an additional hard link (ln, page 103) to a file, you create another reference (an additional filename) to the inode that describes the file. You do not create a new inode. Although HFS+ implements hard links through a different mechanism, their appearance is the same as on a traditional UNIX filesystem.

When you remove (rm) a file, you delete the directory entry that describes the file. When you remove the last hard link to a file, the operating system puts all the blocks the inode pointed to back in the free list (the list of blocks on the disk that are available for use) and frees the inode to be used again.

Symbolic links

Because each filesystem has a separate set of inodes or directory entries, you can create hard links to a file only from within the filesystem that holds that file. To get around this limitation, Mac OS X provides symbolic links, which are files that point to other files. Files that are linked by a symbolic link do not share an inode: You can create a symbolic link to a file from any filesystem. You can also create a symbolic link to a directory, device, or other special file. For more information refer to "Symbolic Links" on page 105.

Special Files

Special files represent Mac OS X kernel routines that provide access to an operating system feature. Device files communicate with a peripheral device. FIFO (first in, first out) special files allow unrelated programs to exchange information. Sockets allow unrelated processes on the same or different systems to exchange information. One type of socket, the UNIX domain socket, is a special file. Symbolic links are another type of special file.

Device Files

Device files, which include block and character special files, represent device drivers that let you communicate with peripheral devices, such as terminals, printers, and hard disks, as well as with virtual devices such as the bit bucket, /dev/null (page 444). By convention, device files appear in the /dev directory and its subdirectories. Each device file represents a device: You read from and write to the file to read from and write to the device it represents. The following example shows part of the output that an ls l command produces for the /dev directory:

$ ls -l /dev
total 4
crw-------   1 root  wheel      23,   0 May 19 16:45 bpf0
crw-------   1 root  wheel      23,   1 May  8 16:44 bpf1
crw-------   1 root  wheel      23,   2 May  8 16:44 bpf2
crw-------   1 root  wheel      23,   3 May  8 16:44 bpf3
crw-------   1 zach  zach        0,   0 May 19 20:50 console
crw-rw-rw-   1 root  wheel       9,   1 May  8 16:45 cu.modem
brw-r-----   1 root  operator   14,   0 May  8 16:44 disk0
br--r-----   1 root  operator   14,   1 May  8 16:44 disk0s1
brw-r-----   1 root  operator   14,   2 May  8 16:44 disk0s3
brw-r-----   1 root  operator   14,   3 May  8 16:44 disk0s5
brw-r-----   1 root  operator   14,   4 May  8 16:44 disk0s6
brw-r-----   1 root  operator   14,   5 May  8 16:44 disk0s7
...

The first character of each line in /dev is either b, c, d, or l, representing a block, character, directory, or symbolic link file, respectively. The next nine characters represent the permissions for the file, followed by the number of hard links and the names of the owner and group. Where the number of bytes would appear for an ordinary or directory file, a device file shows its major and minor device numbers (next) separated by a comma. The rest of the line is the same as any other ls l listing (page 87).

Major and Minor Device Numbers

A major device number represents a class of hardware devices: terminal, printer, hard disk, and so on. In the preceding /dev directory listing, all of the hard disk partitions have a major device number of 14.

A minor device number represents a particular piece of hardware within a class. Although all of the hard disk partitions are grouped together by their major device number, each has a different minor device number (disk0s1 is 1, disk0s2 is 2, and so on). This setup allows one piece of software (the device driver) to service all similar hardware and to be able to distinguish among different physical units.

Block and Character Devices

A block device is a device that handles data in fixed-size chunks, called blocks. An example is a disk partition (a volume). A character device is any device that is not a block device. Examples of a character device include printers, terminals, and modems.

FIFO Special Files (Named Pipes)

A FIFO special file, also called a named pipe, represents a pipe: You write to and read from the file to write to and read from the pipe. The term FIFO stands for first in, first outthe way any pipe works. The first information that you put in one end is the first information that comes out the other end. When you use a pipe on the command line to send the output of a program to the printer, the printer prints the information in the same order that the program produced it and sent it into the pipe.

Unless you are writing sophisticated programs, you will not be working with FIFO special files. However, programs that you run under Mac OS X may use named pipes for interprocess communication. You can create a pipe using mkfifo:

$ mkfifo AA
$ ls -l AA
prw-r--r--  1 zach  zach  0 Apr 26 13:11 AA

The p at the left end of the output of ls l indicates that the file is a pipe.

UNIX and related systems have had pipes for many generations. Without named pipes, only processes that were children of the same ancestor could use pipes to exchange information. Using named pipes, any two processes on a single system can exchange information. Suppose one program writes to a FIFO special file and another program reads from the same file. The two programs do not have to run at the same time or be aware of each other's activity. The operating system handles all buffering and information storage. The term asynchronous (async) applies to this type of communication because programs on the ends of the pipe do not have to be synchronized.

Sockets

Like a FIFO special file, a socket allows asynchronous processes that are not children of the same ancestor to exchange information. Sockets are the central mechanism of the interprocess communication that is the basis of the networking facility. When you use networking utilities, pairs of cooperating sockets manage the communication between the processes on the local and remote systems. Sockets form the basis of utilities such as ssh and scp.


Links

A link is a pointer to a file. Every time you create a file by using vim or cp or any other means, you are putting a pointer in a directory. This pointer associates a filename with a place on the disk. When you specify a filename in a command, you are indirectly pointing to the place on the disk that holds the information you want.

Sharing files can be useful when two or more people are working on the same project and need to share some information. You can make it easy for other users to access one of your files by creating additional links to the file.

To share a file with another user, first give the user permission to read from and write to the file (page 88). You may also have to change the access permission of the parent directory of the file to give the user read, write, or execute permission (page 90). Once the permissions are appropriately set, the user can create a link to the file so that each of you can access the file from your separate directory hierarchies.

A link can also be useful to a single user with a large directory hierarchy. You can create links to cross-classify files in your directory hierarchy, using different classifications for different tasks. For example, if you have the file layout depicted in Figure 4-2, you might have a file named to_do in each subdirectory of the correspond directorythat is, in personal, memos, and business. If you find it difficult to keep track of everything you need to do, you can create a separate directory named to_do in the correspond directory. You can then link each subdirectory's to-do list into that directory. For example, you could link the file named to_do in the memos directory to a file named memos in the to_do directory. This set of links is shown in Figure 4-12.

Figure 4-12. Using links to cross-classify files

Although it may sound complicated, this technique keeps all of your to-do lists conveniently in one place. The appropriate list is easily accessible in the task-related directory when you are busy composing letters, writing memos, or handling personal business.

Tip: About the discussion of hard links

Two kinds of links exist: hard links and symbolic (soft) links. Hard links are older and becoming outdated. The section on hard links is marked as optional; you can skip it, although it discusses inodes and gives you insight into the structure of the filesystem.

Optional: Hard Links

A hard link to a file appears as another file. If the file appears in the same directory as the linked-to file, the links must have different filenames because two files in the same directory cannot have the same name. You can create a hard link to a file only from within the filesystem that holds that file.

ln: CREATES A HARD LINK

The ln (link) utility (without the s [symbolic] option) creates an additional hard link to an existing file using the following syntax:

ln existing-file new-link

The next command makes the link shown in Figure 4-13 by creating a new link named /Users/zach/letter to an existing file named draft in Max's home directory:

Figure 4-13. Two links to the same file: /Users/zach/letter and /Users/max/draft

$ pwd
/Users/max
$ ln draft /Users/zach/letter

The new link appears in the /Users/zach directory with the filename letter. In practice, Zach may need to change directory and file permissions for Max to be able to access the file.

The ln utility creates an additional pointer to an existing file but it does not make another copy of the file. Because there is only one file, the file status informationsuch as access permissions, owner, and the time the file was last modifiedis the same for all links; only the filenames differ. When Max modifies /Users/max/draft, for example, Zach sees the changes in /Users/zach/letter.

cp VERSUS ln

The following commands verify that ln does not make an additional copy of a file. Create a file, use ln to make an additional link to the file, change the contents of the file through one link, and verify the change through the other link:

$ cat file_a
This is file A.
$ ln file_a file_b
$ cat file_b
This is file A.
$ vim file_b
...
$ cat file_b
This is file B after the change.
$ cat file_a
This is file B after the change.

If you try the same experiment using cp instead of ln and change a copy of the file, the difference between the two utilities will become clearer. Once you change a copy of a file, the two files are different:

$ cat file_c
This is file C.
$ cp file_c file_d
$ cat file_d
This is file C.
$ vim file_d
...
$ cat file_d
This is file D after the change.
$ cat file_c
This is file C.

ls and link counts

You can use ls with the l option, followed by the names of the files you want to compare, to confirm that the status information is the same for two links to the same file and is different for files that are not linked. In the following example, the 2 in the links field (just to the left of zach) shows there are two links to file_a and file_b:

$ ls -l file_a file_b file_c file_d
-rw-r--r-- 2 zach  zach  33  May 24 10:52 file_a
-rw-r--r-- 2 zach  zach  33  May 24 10:52 file_b
-rw-r--r-- 1 zach  zach  16  May 24 10:55 file_c
-rw-r--r-- 1 zach  zach  33  May 24 10:57 file_d

Although it is easy to guess which files are linked to one another in this example, ls does not explicitly tell you.

ls and inodes

Use ls with the i option to determine without a doubt which files are linked. The i option lists the inode (page 937) number for each file. An inode is the control structure for a file. (HFS+ filesystems do not have inodes but, through an elaborate scheme, appear to have inodes.) If the two filenames have the same inode number, they share the same control structure and are links to the same file. Conversely, when two filenames have different inode numbers, they are different files. The following example shows that file_a and file_b have the same inode number and that file_c and file_d have different inode numbers:

$ ls -i file_a file_b file_c file_d
3534 file_a    3534 file_b    5800 file_c    7328 file_d

All links to a file are of equal value: The operating system cannot distinguish the order in which multiple links were created. When a file has two links, you can remove either one and still access the file through the remaining link. You can remove the link used to create the file and, as long as one link remains, still access the file through that link.

Symbolic Links

In addition to hard links, Mac OS X supports symbolic links, also called soft links or symlinks. A hard link is a pointer to a file (the directory entry points to the inode), whereas a symbolic link is an indirect pointer to a file (the directory entry contains the pathname of the pointed-to filea pointer to the hard link to the file).

Advantages of symbolic links

Symbolic links were developed because of the limitations inherent in hard links. You cannot create a hard link to a directory, but you can create a symbolic link to a directory.

Often the Mac OS X directory hierarchy is composed of several filesystems (volumes). Because each filesystem keeps separate control information (that is, separate inode tables or filesystem structures) for the files it holds, it is not possible to create hard links between files in different filesystems. A symbolic link can point to any file, regardless of where it is located in the file structure, whereas a hard link to a file must be in the same filesystem as the other hard link(s) to the file. When you create links only among files in your own directories, you will not notice this limitation.

A major advantage of a symbolic link is that it can point to a nonexistent file. This ability is useful if you need a link to a file that is periodically removed and re-created. A hard link keeps pointing to a "removed" file, which the link keeps alive even after a new file is created. In contrast, a symbolic link always points to the newly created file and does not interfere when you delete the old file. For example, a symbolic link could point to a file that gets checked in and out under a source code control system, a .o file that is re-created by the C compiler each time you run make, or a log file that is periodically archived.

Although they are more general than hard links, symbolic links have some disadvantages. Whereas all hard links to a file have equal status, symbolic links do not have the same status as hard links. When a file has multiple hard links, it is analogous to a person having multiple full legal names, as many married women do. In contrast, symbolic links are analogous to nicknames. Anyone can have one or more nicknames, but these nicknames have a lesser status than legal names. The following sections describe some of the peculiarities of symbolic links.

Finder aliases

A symbolic link is very similar to a Windows shortcut or a Macintosh Finder alias. However, a Finder alias is not a symbolic link and does not work with most shell commands. Conversely, a symbolic link is recognized by the Finder as though it were a Finder alias. A Finder alias is in some ways more similar to a hard link; a Finder alias points to a file, no matter where it is, rather than to a path, no matter what file is stored there.

ln: Creates a Symbolic Link

You use ln with the s option to create a symbolic link. The following example creates a symbolic link /tmp/s3 to the file sum in Zach's home directory. When you use an ls l command to look at the symbolic link, ls displays the name of the link and the name of the file it points to. The first character of the listing is l (for link).

$ ln -s /Users/zach/sum /tmp/s3
$ ls -l /Users/zach/sum /tmp/s3
-rw-rw-r--  1 zach  zach  13 Jun 12 09:51 /Users/zach/sum
lrwxr-xr-x  1 zach  wheel  15 Jun 12 10:09 /tmp/s3 -> /Users/zach/sum
$ cat /tmp/s3
This is sum.

Tip: Use absolute pathnames with symbolic links

Symbolic links are literal and are not aware of directories. A link that points to a relative pathname, including a simple filename, assumes that the relative pathname is relative to the directory that the link was created in (not the directory the link was created from). In the following example, the link points to the file named sum in the /tmp directory. Because no such file exists, cat displays an error message.

$ pwd
/Users/zach
$ ln -s sum /tmp/s4
$ ls -l sum /tmp/s4
lrwxr-xr-x 1 zach  zach   3 Jun 12 10:13 /tmp/s4 -> sum
-rw-rw-r-- 1 zach  wheel   38 Jun 12 09:51 sum
$ cat /tmp/s4
cat: /tmp/s4: No such file or directory

The sizes and times of the last modifications of the two files are different. Unlike a hard link, a symbolic link to a file does not have the same status information as the file itself.

Similarly you can use ln to create a symbolic link to a directory. When you use the s option, ln does not care whether the file you are creating a link to is an ordinary file or a directory.

Optional: cd and Symbolic Links

When you use a symbolic link as an argument to cd to change directories, the results can be confusing, particularly if you did not realize that you were using a symbolic link.

If you use cd to change to a directory that is represented by a symbolic link, the pwd shell builtin lists the name of the symbolic link. The pwd utility (/bin/pwd) lists the name of the linked-to directory, not the link, regardless of how you got there.

$ ln -s /Users/zach/grades /tmp/grades.old
$ pwd
/Users/zach
$ cd /tmp/grades.old
$ pwd
/tmp/grades.old
$ /bin/pwd
/Users/zach/grades

When you change directories back to the parent, you end up in the directory holding the symbolic link. Because /tmp is itself a symbolic link to /private/tmp, after you give a cd .. command, your working directory is /private/tmp.

$ cd ..
$ pwd
/tmp
$ /bin/pwd
/private/tmp

rm: Removes a Link

When you create a file, there is one hard link to it. You can delete the file or, using UNIX terminology, remove the link with the rm utility. When you remove the last hard link to a file, you can no longer access the information stored there and the operating system releases the space the file occupied on the disk for subsequent use by other files. The space is released even if symbolic links to the file remain. When there is more than one hard link to a file, you can remove a hard link and still access the file from any remaining link. Unlike dragging a file to the Finder's Trash folder, this operation cannot be undone. A skilled hacker can sometimes piece the file together with time and effort.

When you remove all hard links to a file, you will not be able to access the file through a symbolic link. In the following example, cat reports that the file total does not exist because it is a symbolic link to a file that has been removed:

$ ls -l sum
-rw-r--r--  1 zach  zach  981 May 24 11:05 sum
$ ln -s sum total
$ rm sum
$ cat total
cat: total: No such file or directory
$ ls l total
lrwxr-xr-x  1 zach  zach  3 May 24 11:09 total -> sum

When you remove a file, be sure to remove all symbolic links to it. Remove a symbolic link in the same way you remove other files:

$ rm total


Chapter Summary

Mac OS X has a hierarchical, or treelike, file structure that makes it possible to organize files so that you can find them quickly and easily. The file structure contains directory files and ordinary files. Directories contain other files, including other directories; ordinary files generally contain text, programs, or images. The ancestor of all files is the root directory named /.

Most OS X filesystems support 255-character filenames. Nonetheless, it is a good idea to keep filenames simple and intuitive. Filename extensions can help make filenames more meaningful.

When you are logged in, you are always associated with a working directory. Your home directory is your working directory from the time you first log in until you use cd to change directories.

An absolute pathname starts with the root directory and contains all the filenames that trace a path to a given file. Such a pathname starts with a slash, representing the root directory, and contains additional slashes between the other filenames in the path.

A relative pathname is similar to an absolute pathname but starts the path tracing from the working directory. A simple filename is the last element of a pathname and is a form of a relative pathname.

Among the attributes associated with each file are access permissions. They determine who can access the file and how the file may be accessed. Three groups of users can potentially access the file: the owner, members of a group, and all other users. An ordinary file can be accessed in three ways: read, write, and execute. The ls utility with the l option displays these permissions. For directories, execute access is redefined to mean that the directory can be searched.

The owner of a file or Superuser can use the chmod utility to change the access permissions of a file. This utility specifies read, write, and execute permissions for the file's owner, the group, and all other users on the system.

Mac OS X files have extended attributes, which include file attributes used by the Finder, Access Control Lists, and resource forks. Not all utilities recognize extended attributes.

An ordinary file stores user data, such as textual information, programs, or an image. A directory is a standard-format disk file that stores information, including names, about ordinary files and other directory files. An inode is a data structure, stored on disk, that defines a file's existence and is identified by an inode number. A directory relates each of the filenames it stores to a specific inode.

A link is a pointer to a file. You can have several links to a single file so that you can share the file with other users or have the file appear in more than one directory. Because only one copy of a file with multiple links exists, changing the file through any one link causes the changes to appear in all the links. Hard links cannot link directories or span filesystems, whereas symbolic links can.

Table 4-4 summarizes the utilities introduced in this chapter.

Table 4-4. Utilities introduced in this chapter

cd

Associates you with another working directory (page 82)

chmod

Changes the access permissions on a file (page 88)

ln

Makes a link to an existing file (page 103)

mkdir

Creates a directory (page 80)

pwd

Displays the pathname of the working directory (page 77)

rmdir

Deletes a directory (page 83)


Exercises

1.

Is each of the following an absolute pathname, a relative pathname, or a simple filename?

  1. milk_co

  2. correspond/business/milk_co

  3. /Users/zach

  4. /Users/zach/literature/promo

  5. ..

  6. letter.0610

2.

List the commands you can use to perform these operations:

  1. Make your home directory the working directory

  2. Identify the working directory

3.

If your working directory is /Users/zach with a subdirectory named literature, give three sets of commands that you can use to create a subdirectory named classics under literature. Also give several sets of commands you can use to remove the classics directory and its contents.

4.

The df utility displays all mounted filesystems along with information about each. Use the df utility with the h (human-readable) option to answer the following questions.

  1. How many filesystems are mounted on your Mac OS X system?

  2. Which filesystem stores your home directory?

  3. Assuming that your answer to exercise 4a is two or more, attempt to create a hard link to a file on another filesystem. What error message do you get? What happens when you attempt to create a symbolic link to the file instead?

5.

Suppose you have a file that is linked to a file owned by another user. What can you do so that changes to the file are no longer shared?

6.

What are some of the differences between a Finder alias and a hard link or a symbolic link?

7.

If /Users/max/draft and /Users/zach/letter are links to the same file and the following sequence of events occurs, what will be the date in the opening of the letter?

  1. Zach gives the command vim letter.

  2. Max gives the command vim draft.

  3. Max changes the date in the opening of the letter to January 31, 2006, writes the file, and exits from vim.

  4. Zach changes the date to February 1, 2006, writes the file, and exits from vim.

8.

Suppose that a user belongs to a group that has all permissions on a file named jobs_list, but the user, as the owner of the file, has no permissions. Describe which operations, if any, the user/owner can perform on jobs_list. Which command can the user/owner give that will grant the user all permissions on the file?

9.

Does the root directory have any files that are hidden in the Finder but displayed by ls? Explain why these files might be hidden.

10.

Assume that you are given the directory structure shown in Figure 4-2 on page 73 and the following directory permissions:

d--x--x---  3 max  max  578 Mar 10 15:16 business
drwxr-xr-x  2 max  max  578 Mar 10 15:16 business/milk_co

For each category of permissionsowner, group, and otherwhat happens when you run each of the following commands? Assume that the working directory is the parent of correspond and that the file cheese_co is readable by everyone.

  1. cd correspond/business/milk_co

  2. ls l correspond/business

  3. cat correspond/business/cheese_co

11.

The FAT16 filesystem, also called the MS-DOS filesystem, is inefficient on large disks and imposes stringent filename limits. Why is it important to support this filesystem?

12.

What does the .. entry in a directory point to? What does this entry point to in the root (/) directory?


Advanced Exercises

13.

How can you create a file named i? Which techniques do not work, and why do they not work? How can you remove the file named i?

14.

Suppose that the working directory contains a single file named andor. What error message do you get when you run the following command line?

$ mv andor and\/or

Under what circumstances is it possible to run the command without producing an error?

15.

The ls i command displays a filename preceded by the inode number of the file (page 104). Write a command to output inode/filename pairs for the files in the working directory, sorted by inode number. (Hint: Use a pipe.)

16.

Explain the error messages displayed in the following sequence of commands:

$ ls -l
total 0
drwxrwxr-x  3 zach  zach  102 Mar  2 17:57 dirtmp
$ ls dirtmp
$ rmdir dirtmp
rmdir: dirtmp: Directory not empty
$ rm dirtmp/*
rm: dirtmp/*: No such file or directory

17.

Is it possible to distinguish a file from a hard link to a file? That is, given a filename, can you tell whether it was created using an ln command? Explain.


Chapter 5. The Shell

Previous Page
Next Page

5. The Shell

IN THIS CHAPTER

The Command Line | 114
Standard Input and Standard Output | 120
Redirection | 122
Pipes | 128
Running a Program in the Background | 131
kill: Aborting a Background Job | 132
Filename Generation/Pathname Expansion | 133
Builtins | 138

This chapter takes a close look at the shell and explains how to use some of its features. For example, it discusses command line syntax and also describes how the shell processes a command line and initiates execution of a program. The chapter also explains how to redirect input to and output from a command, construct pipes and filters on the command line, and run a command in the background. The final section covers filename expansion and explains how you can use this feature in your everyday work.

Except as noted everything in this chapter applies to the Bourne Again (bash) and TC (tcsh) Shells. The exact wording of the shell output differs from shell to shell: What your shell displays may differ slightly from what appears in this book. For shell-specific information, refer to Chapters 8 (bash) and 9 (tcsh). Chapter 13 covers writing and executing bash shell scripts.


The Command Line

The shell executes a program when you give it a command in response to its prompt. For example, when you give the ls command, the shell executes the utility program named ls. You can cause the shell to execute other types of programssuch as shell scripts, application programs, and programs you have writtenin the same way. The line that contains the command, including any arguments, is called the command line. In this book the term command refers to the characters you type on the command line as well as to the program that action invokes.

Syntax

Command line syntax dictates the ordering and separation of the elements on a command line. When you press the RETURN key after entering a command, the shell scans the command line for proper syntax. The syntax for a basic command line is

command [arg1] [arg2] ... [argn] RETURN

One or more SPACEs must separate elements on the command line. The command is the name of the command, arg1 through argn are arguments, and RETURN is the keystroke that terminates all command lines. The brackets in the command line syntax indicate that the arguments they enclose are optional. Not all commands require arguments: Some commands do not allow arguments; other commands allow a variable number of arguments; and others require a specific number of arguments. Options, a special kind of argument, are usually preceded by one or two hyphens (also called a dash or minus sign: -).

Command Name

Usage message

Some useful Mac OS X command lines consist of only the name of the command without any arguments. For example, ls by itself lists the contents of the working directory. Most commands accept one or more arguments. Commands that require arguments typically give a short error message, called a usage message, when you use them without arguments, with incorrect arguments, or with the wrong number of arguments.

Arguments

On the command line each sequence of nonblank characters is called a token or word. An argument is a token, such as a filename, string of text, number, or other object that a command acts on. For example, the argument to a vim or emacs command is the name of the file you want to edit.

The following command line shows cp copying the file named temp to tempcopy:

$ cp temp tempcopy

Arguments are numbered starting with the command itself as argument zero. In this example cp is argument zero, temp is argument one, and tempcopy is argument two. The cp utility requires two arguments on the command line. (The utility can take more arguments but not fewer; see page 690.) Argument one is the name of an existing file. Argument two is the name of the file that cp is creating or overwriting. Here the arguments are not optional; both arguments must be present for the command to work. When you do not supply the right number or kind of arguments, cp displays a usage message. Try typing cp and then pressing RETURN.

Options

An option is an argument that modifies the effects of a command. You can frequently specify more than one option, modifying the command in several different ways. Options are specific to and interpreted by the program that the command line calls, not the shell.

By convention options are separate arguments that follow the name of the command and usually precede other arguments, such as filenames. Most utilities require you to prefix options with a single hyphen. However, this requirement is specific to the utility and not the shell. GNU program options are frequently preceded by two hyphens in a row. For example, --help generates a (sometimes extensive) usage message.

Figure 5-1 first shows what happens when you give an ls command without any options. By default ls lists the contents of the working directory in alphabetical order, vertically sorted in columns. Next the -r (reverse order) option causes the ls utility to display the list of files in reverse alphabetical order, still sorted in columns. The -x option causes ls to display the list of files in horizontally sorted rows.

Figure 5-1. Using options
$ ls
alex         jenny        office       temp
hold         mark         oldstuff     test
house        names        personal
$ ls -r
test         oldstuff     mark         hold
temp         office       jenny        alex
personal     names        house
$ ls -x
alex         hold         house        jenny        mark
names        office       oldstuff     personal     temp
test
$ ls -rx
test         temp         personal     oldstuff     office
names        mark         jenny        house        hold
alex

Combining options

When you need to use several options, you can usually group multiple single-letter options into one argument that starts with a single hyphen; do not put SPACEs between the options. You cannot combine options that are preceded by two hyphens in this way, however. Specific rules for combining options depend on the program you are running. Figure 5-1 shows both the -r and -x options with the ls utility. Together these options generate a list of filenames in horizontally sorted columns, in reverse alphabetical order. Most utilities allow you to list options in any order; thus ls -xr produces the same results as ls -rx. The command ls -x -r also generates the same list.

Tip: Displaying readable file sizes: the -h option

Most utilities that report on file sizes specify the size of a file in bytes. Bytes work well when you are dealing with smaller files, but the numbers can be difficult to read when you are working with file sizes that are measured in megabytes or gigabytes. Use the -h (human readable) option to display file sizes in kilo-, mega-, and gigabytes. Experiment with df -h (disk free) and ls -lh commands.

Option arguments

Some utilities have options that themselves require arguments. For example, the gcc utility has a -o option that must be followed by the name you want to give the executable file that gcc generates. Typically an argument to an option is separated from its option letter by a SPACE:

$ gcc -o prog prog.c

Arguments that start with a hyphen

Another convention allows utilities to work with arguments, such as filenames, that start with a hyphen. If a file's name is -l, the following command is ambiguous:

$ ls -l

This command could mean a long listing of all files in the working directory or a listing of the file named -l. It is interpreted as the former. You should avoid creating files whose names begin with hyphens. If you do create them, many utilities follow the convention that a -- argument (two consecutive hyphens) indicates the end of the options (and the beginning of the arguments). To disambiguate the command, you can type

$ ls -- -l

You can use an alternative format in which the period refers to the working directory and the slash indicates that the name refers to a file in the working directory:

$ ls ./-l

Assuming that you are working in the /Users/zach directory, the preceding command is functionally equivalent to

$ ls /Users/zach/-l

You can give the following command to get a long listing of this file:

$ ls -l -- -l

These are conventions, not hard-and-fast rules, and a number of utilities do not follow them (e.g., find). Following such conventions is a good idea; it makes it much easier for users to work with your program. When you write shell programs that require options, follow the standard option conventions.

Tip: The GNU --help option

Many utilities display a (sometimes extensive) help message when you call them with an argument of --help. All utilities developed by the GNU project (page 4) accept this option. An example follows.

$ bzip2 --help
bzip2, a block-sorting file compressor.  Version 1.0.2, 30-Dec-2001.

   usage: bzip2 [flags and input files in any order]

   -h --help           print this message
   -d --decompress     force decompression
   -z --compress       force compression
   -k --keep           keep (don't delete) input files
   -f --force          overwrite existing output files
   -t --test           test compressed file integrity
   -c --stdout         output to standard out
   -q --quiet          suppress noncritical error messages
   -v --verbose        be verbose (a 2nd -v gives more)
...

Processing the Command Line

As you enter a command line, the Mac OS X tty device driver (part of the Mac OS X operating system kernel) examines each character to see whether it must take immediate action. When you press DELETE (to erase a character) or CONTROL-U (to kill a line), the device driver immediately adjusts the command line as required; the shell never sees the character(s) you erased or the line you killed. Often a similar adjustment occurs when you press CONTROL-W (to erase a word). When the character you entered does not require immediate action, the device driver stores the character in a buffer and waits for additional characters. When you press RETURN, the device driver passes the command line to the shell for processing.

Parsing the command line

When the shell processes a command line, it looks at the line as a whole and parses (breaks) it into its component parts (Figure 5-2). Next the shell looks for the name of the command. Usually the name of the command is the first item on the command line after the prompt (argument zero). The shell takes the first characters on the command line up to the first blank (TAB or SPACE) and then looks for a command with that name. The command name (the first token) can be specified on the command line either as a simple filename or as a pathname. For example, you can call the ls command in either of the following ways:

$ ls
$ /bin/ls

Figure 5-2. Processing the command line

Optional

The shell does not require that the name of the program appear first on the command line. Thus you can structure a command line as follows:

$ >bb <aa cat

This command runs cat with standard input coming from the file named aa and standard output going to the file named bb. When the shell recognizes the redirect symbols (page 122), it recognizes and processes them and their arguments before finding the name of the program that the command line is calling. This is a properly structuredalbeit rarely encountered and possibly confusingcommand line.

Absolute versus relative pathnames

When you give an absolute pathname on the command line or a relative pathname that is not a simple filename (i.e., any pathname that includes at least one slash), the shell looks in the specified directory (/bin in the case of the /bin/ls command) for a file that has the name ls and that you have permission to execute. When you give a simple filename, the shell searches through a list of directories for a filename that matches the specified name and that you have execute permission for. The shell does not look through all directories but only the ones specified by the variable named PATH. Refer to page 285 (bash) or page 359 (tcsh) for more information on PATH. Also refer to the discussion of the which and type utilities on page 58.

When it cannot find the executable file, the Bourne Again Shell (bash) displays a message such as the following:

$ abc
bash: abc: command not found

One reason the shell may not be able to find the executable file is that it is not in a directory in your PATH. Under bash the following command temporarily adds the working directory (.) to your PATH:

$ PATH=$PATH:.

For security reasons, you may not want to add the working directory to PATH permanently; see the adjacent tip and the one on page 286.

Tip: Try giving a command as ./command

You can always execute an executable file in the working directory by prepending ./ to the name of the file. For example, if toad is an executable file in the working directory, you can execute it with the following command:

$ ./toad

When the shell finds the program but cannot execute it (you do not have execute permission for the file that contains the program), it displays a message similar to

$ def
bash: ./def: Permission denied

See "ls -l: Displays Permissions" on page 87 for information on displaying access permissions for a file and "chmod: Changes Access Permissions" on page 88 for instructions on how to change file access permissions.

Executing the Command Line

Process

If it finds an executable file with the same name as the command, the shell starts a new process. A process is the execution of a command by Mac OS X (page 293). The shell makes each command line argument, including options and the name of the command, available to the called program. While the command is executing, the shell waits for the process to finish. At this point the shell is in an inactive state called sleep. When the program finishes execution, it passes its exit status (page 564) to the shell. The shell then returns to an active state (wakes up), issues a prompt, and waits for another command.

The shell does not process arguments

Because the shell does not process command line arguments but only hands them to the called program, the shell has no way of knowing whether a particular option or other argument is valid for a given program. Any error or usage messages about options or arguments come from the program itself. Some utilities ignore bad options.


Standard Input and Standard Output

Standard output is a place that a program can send information, such as text. The program never "knows" where the information it sends to standard output is going (Figure 5-3). The information can go to a printer, an ordinary file, or the screen.

Figure 5-3. The command does not know where standard input comes from or where standard output and standard error go

The following sections show that by default the shell directs standard output from a command to the screen[1] and describe how you can cause the shell to redirect this output to another file.

[1] The term screen is used throughout this book to mean screen, terminal emulator window, or workstation. Screen refers to the device that you see the prompt and messages displayed on.

Standard input is a place that a program gets information from. As with standard output the program never "knows" where the information came from. The following sections also explain how to redirect standard input to a command so that it comes from an ordinary file instead of from the keyboard (the default).

In addition to standard input and standard output, a running program normally has a place to send error messages: standard error. Refer to page 262 (bash) and page 346 (tcsh) for more information on handling standard error.

The Screen as a File

Chapter 4 introduced a number of types of files, including device files. A device file resides in the Mac OS X file structure, usually in the /dev directory, and represents a peripheral device, such as a terminal emulator window, screen, printer, or disk drive.

The device name that the who utility displays after your login name is the filename of your screen. For example, when who displays the device name ttyp4, the pathname of your screen is /dev/ttyp4. When you work with multiple windows, each window has its own device name. You can also use the tty utility to display the name of the device that you give the command from. Although you would not normally have occasion to do so, you can read from and write to this file as though it were a text file. Writing to it displays what you wrote on the screen; reading from it reads what you entered on the keyboard.

Tip: chsh: changes your login shell

The person who sets up your account determines which shell you will use when you first log in on the system or when you open a terminal emulator window in a GUI environment. You can run any shell you like once you are logged in. Enter the name of the shell you want to use (bash or tcsh) and press RETURN; the next prompt will be that of the new shell. Give an exit command to return to the previous shell. Because shells you call in this manner are nested (one runs on top of the other), you will be able to log out only from your original shell. When you have nested several shells, keep giving exit commands until you reach your original shell. You will then be able to log out.

Use the chsh utility when you want to change your login shell permanently. First give the command chsh. The chsh utility opens in an editor (usually vim) a temporary file describing your account and listing your shell. Enter the absolute pathname of the shell you want to use (/bin/bash or /bin/tcsh) and save the file. When you change your login shell in this manner using a terminal emulator under a GUI, subsequent terminal emulator windows will not reflect the change until you log out of the system and log back in.

The Keyboard and Screen as Standard Input and Standard Output

When you first log in, the shell directs standard output of your commands to the device file that represents your screen (Figure 5-4). Directing output in this manner causes it to appear on your screen. The shell also directs standard input to come from the same file, so that your commands receive as input anything you type on your keyboard.

Figure 5-4. By default standard input comes from the keyboard and standard output goes to the screen

cat

The cat utility provides a good example of the way the keyboard and the screen function as standard input and standard output, respectively. When you use cat, it copies a file to standard output. Because the shell directs standard output to the screen, cat displays the file on the screen.

Up to this point cat has taken its input from the filename (argument) you specified on the command line. When you do not give cat an argument (that is, when you give the command cat followed immediately by RETURN), cat takes its input from standard input. Thus, when called without an argument, cat copies standard input to standard output, one line at a time.

To see how cat works, type cat and press RETURN in response to the shell prompt. Nothing happens. Enter a line of text and press RETURN. The same line appears just under the one you entered. The cat utility is working. Because the shell associates cat's standard input with the keyboard and cat's standard output with the screen, when you type a line of text cat copies the text from standard input (the keyboard) to standard output (the screen). This exchange is shown in Figure 5-5.

Figure 5-5. The cat utility copies standard input to standard output
$ cat
This is a line of text.
This is a line of text.
Cat keeps copying lines of text
Cat keeps copying lines of text
until you press CONTROL-D at the beginning
until you press CONTROL-D at the beginning
of a line.
of a line.
CONTROL-D
$

CONTROL-D signals EOF

The cat utility keeps copying text until you enter CONTROL-D on a line by itself. Pressing CONTROL-D sends an EOF (end of file) signal to cat to indicate that it has reached the end of standard input and there is no more text for it to copy. The cat utility then finishes execution and returns control to the shell, which displays a prompt.

Redirection

The term redirection encompasses the various ways you can cause the shell to alter where standard input of a command comes from and where standard output goes to. By default the shell associates standard input and standard output of a command with the keyboard and the screen as mentioned earlier. You can cause the shell to redirect standard input or standard output of any command by associating the input or output with a command or file other than the device file representing the keyboard or the screen. This section demonstrates how to redirect input from and output to ordinary text files and utilities.

Redirecting Standard Output

The redirect output symbol (>) instructs the shell to redirect the output of a command to the specified file instead of to the screen (Figure 5-6). The format of a command line that redirects output is

command [arguments] > filename

Figure 5-6. Redirecting standard output

where command is any executable program (such as an application program or a utility), arguments are optional arguments, and filename is the name of the ordinary file the shell redirects the output to.

Figure 5-7 uses cat to demonstrate output redirection. This figure contrasts with Figure 5-3 on page 120, where both standard input and standard output are associated with the keyboard and the screen. The input in Figure 5-7 comes from the keyboard. The redirect output symbol on the command line causes the shell to associate cat's standard output with the sample.txt file specified on the command line.

Figure 5-7. cat with its output redirected
$ cat > sample.txt
This text is being entered at the keyboard and
cat is copying it to a file.
Press CONTROL-D to indicate the
end of file.
CONTROL-D
$

After giving the command and typing the text shown in Figure 5-7, the sample.txt file contains the text you entered. You can use cat with an argument of sample.txt to display this file. The next section shows another way to use cat to display the file.

Caution: Redirecting output can destroy a file I

Use caution when you redirect output to a file. If the file exists, the shell will overwrite it and destroy its contents. For more information see the "Redirecting output can destroy a file II" caution on page 126.

Figure 5-7 shows that redirecting the output from cat is a handy way to create a file without using an editor. The drawback is that once you enter a line and press RETURN, you cannot edit the text. While you are entering a line, the erase and kill keys work to delete text. This procedure is useful for making short, simple files.

Figure 5-8 shows how to use cat and the redirect output symbol to catenate (join one after the otherthe derivation of the name of the cat utility) several files into one larger file. The first three commands display the contents of three files: stationery, tape, and pens. The next command shows cat with three filenames as arguments. When you call it with more than one filename, cat copies the files, one at a time, to standard output. In this case standard output is redirected to the file supply_orders. The final cat command shows that supply_orders contains the contents of all three files.

Figure 5-8. Using cat to catenate files
$ cat stationery
2,000 sheets letterhead ordered:    10/7/05
$ cat tape
1 box masking tape ordered:         10/14/05
5 boxes filament tape ordered:      10/28/05
$ cat pens
12 doz. black pens ordered:         10/4/05

$ cat stationery tape pens > supply_orders

$ cat supply_orders
2,000 sheets letterhead ordered:    10/7/05
1 box masking tape ordered:         10/14/05
5 boxes filament tape ordered:      10/28/05
12 doz. black pens ordered:         10/4/05

Redirecting Standard Input

Just as you can redirect standard output, so you can redirect standard input. The redirect input symbol (<) instructs the shell to redirect a command's input to come from the specified file instead of from the keyboard (Figure 5-9). The format of a command line that redirects input is

command [arguments] < filename

where command is any executable program (such as an application program or a utility), arguments are optional arguments, and filename is the name of the ordinary file the shell redirects the input from.

Figure 5-9. Redirecting standard input

Figure 5-10 shows cat with its input redirected from the supply_orders file that was created in Figure 5-8 and standard output going to the screen. This setup causes cat to display the sample file on the screen. The system automatically supplies an EOF (end of file) signal at the end of an ordinary file.

Figure 5-10. cat with its input redirected
$ cat < supply_orders
2,000 sheets letterhead ordered:     10/7/05
1 box masking tape ordered:          10/14/05
5 boxes filament tape ordered:       10/28/05
12 doz. black pens ordered:          10/4/05

Utilities that take input from a file or standard input

Giving a cat command with input redirected from a file yields the same result as giving a cat command with the filename as an argument. The cat utility is a member of a class of Mac OS X utilities that function in this manner. Other members of this class of utilities include lpr, sort, and grep. These utilities first examine the command line that you use to call them. If you include a filename on the command line, the utility takes its input from the file you specify. If you do not specify a filename, the utility takes its input from standard input. It is the utility or programnot the shell or operating systemthat functions in this manner.

noclobber: Avoids Overwriting Files

The shell provides a feature called noclobber that stops you from inadvertently overwriting an existing file using redirection. When you enable this feature by setting the noclobber variable and then attempt to redirect output to an existing file, the shell displays an error message and does not execute the command. If the preceding examples result in one of the following messages, the noclobber feature has been set. The following examples set noclobber, attempt to redirect the output from echo into an existing file, and then unset noclobber under bash and tcsh:

bash

$ set -o noclobber
$ echo "hi there" > tmp
bash: tmp: Cannot overwrite existing file
$ set +o noclobber
$ echo "hi there" > tmp
$

tcsh

tcsh $ set noclobber
tcsh $ echo "hi there" > tmp
tmp: File exists.
tcsh $ unset noclobber
tcsh $ echo "hi there" > tmp
$

You can override noclobber by putting a pipe symbol (tcsh uses an exclamation point) after the symbol you use for redirecting output (>|).

In the following example, the user first creates a file named a by redirecting the output of date to the file. Next the user sets the noclobber variable and tries redirecting output to a again. The shell returns an error message. Then the user tries the same thing but using a pipe symbol after the redirect symbol. This time the shell allows the user to overwrite the file. Finally, the user unsets noclobber (using a plus sign in place of the hyphen) and verifies that it is no longer set.

$ date > a
$ set -o noclobber
$ date > a
bash: a: Cannot overwrite existing file
$ date >| a
$ set +o noclobber
$ date > a

For more information on using noclobber under tcsh, refer to page 363.

Caution: Redirecting output can destroy a file II

Depending on which shell you are using and how your environment has been set up, a command such as the following may give you undesired results:

$ cat orange pear > orange
cat: orange: input file is output file

Although cat displays an error message, the shell goes ahead and destroys the contents of the existing orange file. The new orange file will have the same contents as pear because the first action the shell takes when it sees the redirection symbol (>) is to remove the contents of the original orange file. If you want to catenate two files into one, use cat to put the two files into a temporary file and then use mv to rename this third file:

$ cat orange pear > temp
$ mv temp orange

What happens in the next example can be even worse. The user giving the command wants to search through files a, b, and c for the word apple and redirect the output from grep (page 46) to the file a.output. Unfortunately the user enters the filename as a output, omitting the period and inserting a SPACE in its place:

$ grep apple a b c > a output
grep: output: No such file or directory

The shell obediently removes the contents of a and then calls grep. The error message may take a moment to appear, giving you a sense that the command is running correctly. Even after you see the error message, it may take a while to realize that you destroyed the contents of a.

Caution: Redirection works with the data fork of a file only

Redirecting output from a command works only with the data fork of a file and not with the resource fork (page 93). For example, under Mac OS X 10.4 and later the following command copies both the resource and data forks of the file pic.jpg:

$ cp pic.jpg pic.bak.jpg

The next command copies only the data fork of the file:

$ cp < pic.jpg > pic.bak.jpg

Appending Standard Output to a File

The append output symbol (>>) causes the shell to add new information to the end of a file, leaving any existing information intact. This symbol provides a convenient way of catenating two files into one. The following commands demonstrate the action of the append output symbol. The second command accomplishes the catenation described in the preceding caution box:

$ cat orange
this is orange
$ cat pear >> orange
$ cat orange
this is orange
this is pear

You first see the contents of the orange file. Next the contents of the pear file is added to the end of (catenated with) the orange file. The final cat shows the result.

Caution: Do not trust noclobber

Appending output is simpler than the two-step procedure described in the preceding caution box but you must be careful to include both greater than signs. If you accidentally use only one and the noclobber feature is not on, you will overwrite the orange file. Even if you have the noclobber feature turned on, it is a good idea to keep backup copies of files you are manipulating in these ways in case you make a mistake.

Although it protects you from making an erroneous redirection, noclobber does not stop you from overwriting an existing file using cp or mv. These utilities include the -i (interactive) option that helps protect you from this type of mistake by verifying your intentions when you try to overwrite a file. For more information see the "cp can destroy a file" tip on page 44.

The next example shows how to create a file that contains the date and time (the output from date), followed by a list of who is logged in (the output from who). The first line in Figure 5-11 redirects the output from date to the file named whoson. Then cat displays the file. Next the example appends the output from who to the whoson file. Finally cat displays the file containing the output of both utilities.

Figure 5-11. Redirecting and appending output
$ date > whoson
$ cat whoson
Wed May  4 13:46:33 CDT 2005
$ who >> whoson
$ cat whoson
Wed May  4 13:46:33 CDT 2005
zach     console  May  2 12:39
zach     ttyp2    May  3 03:32

/dev/null: Making Data Disappear

The /dev/null device is a data sink, commonly referred to as a bit bucket. You can redirect output that you do not want to keep or see to /dev/null. The output disappears without a trace:

$ echo "hi there" > /dev/null
$

When you read from /dev/null, you get a null string. Give the following cat command to truncate a file named messages to zero length while preserving the ownership and permissions of the file:

$ ls -l messages
-rw-r--r--   1 alex alex 25315 Oct 24 10:55 messages
$ cat /dev/null > messages
$ ls -l messages
-rw-r--r--   1 alex alex 0 Oct 24 11:02 messages

Pipes

The shell uses a pipe to connect standard output of one command directly to standard input of another command. A pipe (sometimes referred to as a pipeline) has the same effect as redirecting standard output of one command to a file and then using that file as standard input to another command. A pipe does away with separate commands and the intermediate file. The symbol for a pipe is a vertical bar (|). The syntax of a command line using a pipe is

command_a [arguments] | command_b [arguments]

The preceding command line uses a pipe to generate the same result as the following group of command lines:

command_a [arguments] > temp
command_b [arguments] < temp
rm temp

In the preceding sequence of commands, the first line redirects standard output from command_a to an intermediate file named temp. The second line redirects standard input for command_b to come from temp. The final line deletes temp. The command using a pipe is not only easier to type, but is generally more efficient because it does not create a temporary file.

tr

You can use a pipe with any of the Mac OS X utilities that accept input either from a file specified on the command line or from standard input. You can also use pipes with commands that accept input only from standard input. For example, the tr (TRanslate) utility (page 879) takes its input from standard input only. In its simplest usage TR has the following format:

tr string1 string2

The tr utility accepts input from standard input and looks for characters that match one of the characters in string1. Upon finding a match, TR translates the matched character in string1 to the corresponding character in string2. (The first character in string1 translates into the first character in string2, and so forth.) The TR utility sends its output to standard output. In both of the following examples, tr displays the contents of the abstract file with the letters a, b, and c translated into A, B, and C, respectively:

$ cat abstract | tr abc ABC

$ tr abc ABC < abstract

The tr utility does not change the contents of the original file; it cannot change the original file because it does not "know" the source of its input.

lpr

The lpr (line printer) utility also accepts input from either a file or standard input. When you type the name of a file following lpr on the command line, it places that file in the print queue. When you do not specify a filename on the command line, lpr takes input from standard input. This feature enables you to use a pipe to redirect input to lpr. The first set of commands in Figure 5-12 shows how you can use ls and lpr with an intermediate file (temp) to send a list of the files in the working directory to the printer. If the temp file exists, the first command overwrites its contents. The second set of commands sends the same list (with the exception of temp) to the printer using a pipe.

Figure 5-12. A pipe
$ ls > temp
$ lpr temp
$ rm temp

or

$ ls | lpr

The commands in Figure 5-13 redirect the output from the who utility to temp and then display this file in sorted order. The sort utility (page 49) takes its input from the file specified on the command line or, when a file is not specified, from standard input and sends its output to standard output. The sort command line in Figure 5-13 takes its input from standard input, which is redirected (<) to come from temp. The output that sort sends to the screen lists the users in sorted (alphabetical) order.

Figure 5-13. Using a temporary file to store intermediate results
$ who > temp
$ sort < temp
max      ttyp1    May  4 13:49 (localhost)
sam      ttyp3    May  4 13:49 (localhost)
zach     console  May  2 12:39
zach     ttyp2    May  3 03:32
$ rm temp

Because sort can take its input from standard input or from a filename on the command line, omitting the < symbol from Figure 5-13 yields the same result.

Figure 5-14 achieves the same result without creating the temp file. Using a pipe the shell redirects the output from who to the input of sort. The sort utility takes input from standard input because no filename follows it on the command line.

Figure 5-14. A pipe doing the work of a temporary file
$ who | sort
max      ttyp1    May  4 13:49 (localhost)
sam      ttyp3    May  4 13:49 (localhost)
zach     console  May  2 12:39
zach     ttyp2    May  3 03:32

When many people are using the system and you want information about only one of them, you can send the output from who to grep (page 46) using a pipe. The grep utility displays the line containing the string you specifyroot in the following example:

$ who | grep 'root'
root     console  Mar 24 05:00

Another way of handling output that is too long to fit on the screen, such as a list of files in a crowded directory, is to use a pipe to send the output through less (see page 42).

$ ls | less

The less utility displays text one screen at a time. To view another screen, press the SPACE bar. To view one more line, press RETURN. Press h for help and q to quit.

Filters

A filter is a command that processes an input stream of data to produce an output stream of data. A command line that includes a filter uses a pipe to connect standard output of one command to the filter's standard input. Another pipe connects the filter's standard output to standard input of another command. Not all utilities can be used as filters.

In the following example, sort is a filter, taking standard input from standard output of who and using a pipe to redirect standard output to standard input of lpr. This command line sends the sorted output of who to the printer:

$ who | sort | lpr

The preceding example demonstrates the power of the shell combined with the versatility of Mac OS X utilities. The three utilities who, sort, and lpr were not specifically designed to work with each other, but they all use standard input and standard output in the conventional way. By using the shell to handle input and output, you can piece standard utilities together on the command line to achieve the results you want.

Caution: Filters do not process resource forks

Filters work on a stream of data, so a utility that runs as a filter will not process resource forks (page 93). The following command creates a partial copy of file, without the resource fork:

$ cat < file > copy_of_file

tee: Sends Output in Two Directions

The tee utility copies its standard input both to a file and to standard output. The utility is aptly named: It takes a single input and sends the output in two directions. In Figure 5-15 the output of who is sent via a pipe to standard input of tee. The tee utility saves a copy of standard input in a file named who.out and also sends a copy to standard output. Standard output of tee goes via a pipe to standard input of grep, which displays lines containing the string sam.

Figure 5-15. Using tee
$ who | tee who.out | grep sam
sam      ttyp3    May  4 13:49 (localhost)
$ cat who.out
max      ttyp1    May  4 13:49 (localhost)
sam      ttyp3    May  4 13:49 (localhost)
zach     console  May  2 12:39
zach     ttyp2    May  3 03:32


Running a Program in the Background

Foreground

In all the examples so far in this book, commands were run in the foreground. When you run a command in the foreground, the shell waits for it to finish before giving you another prompt and allowing you to continue. When you run a command in the background, you do not have to wait for the command to finish before you start running another command.

Jobs

A job is a series of one or more commands that can be connected by pipes. You can have only one foreground job in a window or on a screen, but you can have many background jobs. By running more than one job at a time, you are using one of Mac OS X's important features: multitasking. Running a command in the background can be useful when the command will run for a long time and does not need supervision. It leaves the screen free so that you can use it for other work. Of course, when you are using a GUI, you can open another window to run another job.

Job number, PID number

To run a command in the background, type an ampersand (&) just before the RETURN that ends the command line. The shell assigns a small number to the job and displays this job number between brackets. Following the job number, the shell displays the process identification (PID) numbera larger number assigned by the operating system. Each of these numbers identifies the command running in the background. Then the shell displays another prompt and you can enter another command. When the background job finishes running, the shell displays a message giving both the job number and the command line used to run the command.

The following examples use the Bourne Again Shell. The TC Shell produces almost identical results. The next example runs in the background and sends its output through a pipe to lpr, which sends it to the printer.

$ ls -l | lpr &
[1] 22092
$

The [1] following the command line indicates that the shell has assigned job number 1 to this job. The 22092 is the PID number of the first command in the job. (The TC Shell shows PID numbers for all commands in the job.) When this background job completes execution, you see the message

[1]+ Done      ls -l | lpr

(If you have set up an alias [page 311] for ls, you will see the alias expansion rather than the command you typed.)

Moving a Job from the Foreground to the Background

CONTROL-Z

You can suspend a foreground job (stop it from running) by pressing the suspend key, usually CONTROL-Z. The shell then stops the process and disconnects standard input from the keyboard. You can put a suspended job in the background and restart it by using the bg command followed by the job number. You do not need to use the job number when there is only one stopped job.

Only the foreground job can take input from the keyboard. To connect the keyboard to a program running in the background, you must bring it into the foreground. Type fg without any arguments when only one job is in the background. When more than one job is in the background, type fg, or a percent sign (%), followed by the number of the job you want to bring into the foreground. The shell displays the command you used to start the job (promptme in the following example), and you can enter any input the program requires to continue:

bash $ fg 1
promptme

Redirect the output of a job you run in the background to keep it from interfering with whatever you are doing on the screen. Refer to "Separating and Grouping Commands" on page 268 for more detail about background tasks.

kill: Aborting a Background Job

The interrupt key (usually CONTROL-C) cannot abort a process you are running in the background; you must use kill (page 761) for this purpose. Follow kill on the command line with either the PID number of the process you want to abort or a percent sign (%) followed by the job number.

Determining a PID number with ps

If you forget the PID number, you can use the ps (process status) utility (page 293) to display it. Using the TC Shell, the following example runs a tail -f outfile command (the -f option causes tail to watch outfile and display new lines as they are written to the file) as a background job, uses ps to display the PID number of the process, and aborts the job with kill. The same commands work under bash. So that it does not interfere with anything on the screen, the message saying that the job is terminated does not appear until you press RETURN after the RETURN that ends the kill command:

tcsh $ tail -f outfile &
[1] 1966
tcsh $ ps | grep tail
 1966  p3 S       0:00.01 tail -f outfile
tcsh $ kill 1966
tcsh $ RETURN
[1]    Terminated           tail -f outfile
tcsh $

If you forget the job number, you can use the jobs command to display a list of job numbers. The next example is similar to the previous one but uses the job number instead of the PID number to kill the job:

tcsh $ tail -f outfile &
[1] 3339
tcsh $ bigjob &
[2] 3340
tcsh $ jobs
[1]  + Running              tail -f outfile
[2]  - Running              bigjob
tcsh $ kill %1
RETURN
[1]    Terminated           tail -f outfile
tcsh $


Filename Generation/Pathname Expansion

Wildcards, globbing

When you give the shell abbreviated filenames that contain special characters, also called metacharacters, the shell can generate filenames that match the names of existing files. These special characters are also referred to as wildcards because they act as the jokers do in a deck of cards. When one of these characters appears in an argument on the command line, the shell expands that argument in sorted order into a list of filenames and passes the list to the program that the command line calls. Filenames that contain these special characters are called ambiguous file references because they do not refer to any one specific file. The process that the shell performs on these filenames is called pathname expansion or globbing.

Ambiguous file references refer to a group of files with similar names quickly, saving you the effort of typing the names individually. They can also help you find a file whose name you do not remember in its entirety. If no filename matches the ambiguous file reference, the shell generally passes the unexpanded referencespecial characters and allto the command.

The ? Special Character

The question mark (?) is a special character that causes the shell to generate filenames. It matches any single character in the name of an existing file. The following command uses this special character in an argument to the lpr utility:

$ lpr memo?

The shell expands the memo? argument and generates a list of files in the working directory that have names composed of memo followed by any single character. The shell then passes this list to lpr. The lpr utility never "knows" that the shell generated the filenames it was called with. If no filename matches the ambiguous file reference, the shell passes the string itself (memo?) to lpr or, if it is set up to do so, passes a null string (see nullglob on page 320).

The following example uses ls first to display the names of all files in the working directory and then to display the filenames that memo? matches:

$ ls
mem   memo12  memo9  memoalex  newmemo5
memo  memo5   memoa  memos
$ ls memo?
memo5  memo9  memoa  memos

The memo? ambiguous file reference does not match mem, memo, memo12, memoalex, or newmemo5. You can also use a question mark in the middle of an ambiguous file reference:

$ ls
7may4report  may4report     mayqreport  may_report
may14report  may4report.79  mayreport   may.report
$ ls may?report
may.report  may4report  may_report  mayqreport

To practice generating filenames, you can use echo and ls. The echo utility displays the arguments that the shell passes to it:

$ echo may?report
may.report may4report may_report mayqreport

The shell first expands the ambiguous file reference into a list of all files in the working directory that match the string may?report and then passes this list to echo, as though you had entered the list of filenames as arguments to echo. Next echo displays the list of filenames.

A question mark does not match a leading period (one that indicates an invisible filename; see page 76). When you want to match filenames that begin with a period, you must explicitly include the period in the ambiguous file reference.

The * Special Character

The asterisk (*) performs a function similar to that of the question mark but matches any number of characters, including zero characters, in a filename. The following example shows all of the files in the working directory and then shows three commands that display all the filenames that begin with the string memo, end with the string mo, and contain the string alx:

$ ls
amemo   memo       memoalx.0620  memosally  user.memo
mem     memo.0612  memoalx.keep  sallymemo
memalx  memoa      memorandum    typescript
$ echo memo*
memo memo.0612 memoa memoalx.0620 memoalx.keep memorandum memosally
$ echo *mo
amemo memo sallymemo user.memo
$ echo *alx*
memalx memoalx.0620 memoalx.keep

The ambiguous file reference memo* does not match amemo, mem, sallymemo, or user.memo. Like the question mark, an asterisk does not match a leading period in a filename.

The -a option causes ls to display invisible filenames. The command echo * does not display . (the working directory), .. (the parent of the working directory), .aaa, or .profile. In contrast, the command echo .* displays only those four names:

$ ls
aaa         memo.sally   sally.0612  thurs
memo.0612   report       saturday
$ ls -a
.   .aaa       aaa         memo.sally  sally.0612  thurs
..  .profile   memo.0612   report      saturday
$ echo *
aaa memo.0612  memo.sally  report sally.0612 saturday thurs
$ echo .*
. .. .aaa .profile

In the following example .p* does not match memo.0612, private, reminder, or report. Next the ls .* command causes ls to list .private and .profile in addition to the contents of the . directory (the working directory) and the .. directory (the parent of the working directory). When called with the same argument, echo displays the names of files (including directories) in the working directory that begin with a dot (.), but not the contents of directories.

$ ls -a
.       .private   memo.0612  reminder
..      .profile   private    report
$ echo .p*
.private .profile
$ ls .*
.private .profile

.:
memo.0612  private    reminder   report

..:
.
.
$ echo .*
. .. .private .profile

You can take advantage of ambiguous file references when you establish conventions for naming files. For example, when you end all text filenames with .txt, you can reference that group of files with *.txt. The next command uses this convention to send all the text files in the working directory to the printer. The ampersand causes lpr to run in the background.

$ lpr * .txt &

The [ ] Special Characters

A pair of brackets surrounding a list of characters causes the shell to match filenames containing the individual characters. Whereas memo? matches memo followed by any character, memo[17a] is more restrictive, and matches only memo1, memo7, and memoa. The brackets define a character class that includes all the characters within the brackets. (GNU calls this a character list; a GNU character class is something different.) The shell expands an argument that includes a character-class definition, by substituting each member of the character class, one at a time, in place of the brackets and their contents. The shell then passes the list of matching filenames to the program it is calling.

Each character-class definition can replace only a single character within a filename. The brackets and their contents are like a question mark that substitutes only the members of the character class.

The first of the following commands lists the names of all the files in the working directory that begin with a, e, i, o, or u. The second command displays the contents of the files named page2.txt, page4.txt, page6.txt, and page8.txt.

$ echo [aeiou]*
...
$ less page[2468].txt
...

A hyphen within brackets defines a range of characters within a character-class definition. For example, [6-9] represents [6789], [a-z] represents all lowercase letters in English, and [a-zA-Z] represents all letters, both uppercase and lowercase, in English.

The following command lines show three ways to print the files named part0, part1, part2, part3, and part5. Each of these command lines causes the shell to call lpr with five filenames:

$ lpr part0 part1 part2 part3 part5

$ lpr part[01235]

$ lpr part[0-35]

The first command line explicitly specifies the five filenames. The second and third command lines use ambiguous file references, incorporating character-class definitions. The shell expands the argument on the second command line to include all files that have names beginning with part and ending with any of the characters in the character class. The character class is explicitly defined as 0, 1, 2, 3, and 5. The third command line also uses a character-class definition but defines the character class to be all characters in the range 0-3 plus 5.

The following command line prints 39 files, part0 through part38:

$ lpr part[0-9] part[12][0-9] part3[0-8]

The next two examples list the names of some of the files in the working directory. The first lists the files whose names start with a through m. The second lists files whose names end with x, y, or z.

$ echo [a-m]*
...
$ echo *[x-z]
...

Optional

When an exclamation point (!) or a caret (^) immediately follows the opening bracket ([) that defines a character class, the string enclosed by the brackets matches any character not between the brackets. Thus [^ab]* matches any filename that does not begin with a or b.

The following examples show that *[^ab] matches filenames that do not end with the letters a or b and that [b-d]* matches filenames that begin with b, c, or d.

$ ls
aa  ab  ac  ad  ba  bb  bc  bd  cc  dd
$ ls *[^ab]
ac  ad  bc  bd  cc  ddcc  dd
$ ls [b-d]*
ba  bb  bc  bd  cc  dd

You can match a hyphen (-) or a closing bracket (]) by placing it immediately before the final closing bracket.

The next example demonstrates that the ls utility cannot interpret ambiguous file references. First ls is called with an argument of ?old. The shell expands ?old into a matching filename, hold, and passes that name to ls. The second command is the same as the first, except the ? is quoted (refer to "Special Characters" on page 40). The shell does not recognize this question mark as a special character and passes it on to ls. The ls utility generates an error message saying that it cannot find a file named ?old (because there is no file named ?old).

$ ls ?old
hold
$ ls \?old
ls: ?old: No such file or directory

Like most utilities and programs, ls cannot interpret ambiguous file references; that work is left to the shell.

Tip: The shell expands ambiguous file references

The shell does the expansion when it processes an ambiguous file reference, not the program that the shell runs. In the examples in this section, the utilities (ls, cat, echo, lpr) never see the ambiguous file references. The shell expands the ambiguous file references and passes a list of ordinary filenames to the utility. In the previous examples, echo shows this to be true because it simply displays its arguments; it never displays the ambiguous file reference.


Builtins

A builtin is a utility (also called a command) that is built into a shell. Each of the shells has its own set of builtins. When it runs a builtin, the shell does not fork a new process. Consequently builtins run more quickly and can affect the environment of the current shell. Because builtins are used in the same way as utilities, you will not typically be aware of whether a utility is built into the shell or is a stand-alone utility.

The echo utility is a shell builtin. The shell always executes a shell builtin before trying to find a command or utility with the same name. See page 570 for an in-depth discussion of builtin commands, page 583 for a list of bash builtins, and page 373 for a list of tcsh builtins.

Listing bash builtins

To get a complete list of bash builtins, give the command info bash builtin. To display a page with more information on each builtin, move the cursor to one of the lines listing a builtin command and press RETURN. Alternatively, after typing info bash, give the command /builtin to search the bash documentation for the string builtin. The cursor will rest on the word Builtin in a menu; press RETURN to display the builtins menu.

Because bash was written by GNU, the info page has better information than does the man page. If you want to read about builtins in the man page, give the command man bash and then search for the section on builtins with the command /^SHELL BUILTIN COMMANDS (search for a line that begins with SHELL . . .).

Listing tcsh builtins

For tcsh, give the command man tcsh to display the tcsh man page and then search for the second occurrence of Builtin commands with the following two commands: /Builtin commands (search for the string) and n (search for the next occurrence of the string).


Chapter Summary

The shell is the Mac OS X command interpreter. It scans the command line for proper syntax, picking out the command name and any arguments. The first argument is argument one, the second is argument two, and so on. The name of the command itself is argument zero. Many programs use options to modify the effects of a command. Most Mac OS X utilities identify an option by its leading one or two hyphens.

When you give it a command, the shell tries to find an executable program with the same name as the command. When it does, the shell executes the program. When it does not, the shell tells you that it cannot find or execute the program. If the command is a simple filename, the shell searches the directories given in the variable PATH in an attempt to locate the command.

When it executes a command, the shell assigns one file to the command's standard input and another file to its standard output. By default the shell causes a command's standard input to come from the keyboard and its standard output to go to the screen. You can instruct the shell to redirect a command's standard input from or standard output to any file or device. You can also connect standard output of one command to standard input of another command using a pipe. A filter is a command that reads its standard input from standard output of one command and writes its standard output to standard input of another command.

When a command runs in the foreground, the shell waits for it to finish before it displays a prompt and allows you to continue. When you put an ampersand (&) at the end of a command line, the shell executes the command in the background and displays another prompt immediately. Run slow commands in the background when you want to enter other commands at the shell prompt. The jobs builtin displays a list of jobs and includes the job number of each.

The shell interprets special characters on a command line to generate filenames. A question mark represents any single character, and an asterisk represents zero or more characters. A single character may also be represented by a character class: a list of characters within brackets. A reference that uses special characters (wildcards) to abbreviate a list of one or more filenames is called an ambiguous file reference.

A builtin is a utility that is built into a shell. Each shell has its own set of builtins. When it runs a builtin, the shell does not fork a new process. Consequently builtins run more quickly and can affect the environment of the current shell.

Utilities and Builtins Introduced in This Chapter

Table 5-1 lists the utilities introduced in this chapter.

Table 5-1. New utilities

Utility

Function

tr

Maps one string of characters into another (page 128)

tee

Sends standard input to both a file and standard output (page 131)

bg

Moves a process into the background (page 132)

fg

Moves a process into the foreground (page 132)

jobs

Displays a list of currently running jobs (page 133)


Exercises

1.

What does the shell ordinarily do while a command is executing? What should you do if you do not want to wait for a command to finish before running another command?

2.

Using sort as a filter, rewrite the following sequence of commands:

$ sort list > temp
$ lpr temp
$ rm temp

3.

What is a PID number? Why are these numbers useful when you run processes in the background? Which utility displays the PID numbers of the commands you are running?

4.

Assume that the following files are in the working directory:

$ ls
intro      notesb     ref2      section1    section3    section4b
notesa     ref1       ref3      section2    section4a   sentrev

Give commands for each of the following, using wildcards to express filenames with as few characters as possible.

  1. List all files that begin with section.

  2. List the section1, section2, and section3 files only.

  3. List the intro file only.

  4. List the section1, section3,ref1, and ref3 files.

5.

Refer to the documentation of utilities in Part V or the man pages to determine which commands will

  1. Output the number of lines in the standard input that contain the word a or A.

  2. Output only the names of the files in the working directory that contain the pattern $(.

  3. List the files in the working directory in their reverse alphabetical order.

  4. Send a list of files in the working directory to the printer, sorted by size.

6.

Give a command to

  1. Redirect the standard output from a sort command into a file named phone_list. Assume that the input file is named numbers.

  2. Translate all occurrences of the characters [ and { to the character (, and all occurrences of the characters ] and } to the character ) in the file named permdemos.c. (Hint: Refer to tr on page 879.)

  3. Create a file named book that contains the contents of two other files: part1 and part2.

7.

The lpr and sort utilities accept input either from a file named on the command line or from standard input.

  1. Name two other utilities that function in a similar manner.

  2. Name a utility that accepts its input only from standard input.

8.

Give an example of a command that uses grep

  1. With both input and output redirected.

  2. With only input redirected.

  3. With only output redirected.

  4. Within a pipe.

  5. In which of the preceding is grep used as a filter?

9.

Explain the following error message. What filenames would a subsequent ls display?

$ ls
abc  abd  abe  abf  abg  abh
$ rm abc ab*
rm: cannot remove 'abc': No such file or directory


Advanced Exercises

10.

When you use the redirect output symbol (>) with a command, the shell creates the output file immediately, before the command is executed. Demonstrate that this is true.

11.

In experimenting with shell variables, Alex accidentally deletes his PATH variable. He decides that he does not need the PATH variable. Discuss some of the problems he may soon encounter and explain the reasons for these problems. How could he easily return PATH to its original value?

12.

Assume that your permissions allow you to write to a file but not to delete it.

  1. Give a command to empty the file without invoking an editor.

  2. Explain how you might have permission to modify a file that you cannot delete.

13.

If you accidentally create a filename that contains a nonprinting character, such as a CONTROL character, how can you rename the file?

14.

Why does the noclobber variable not protect you from overwriting an existing file with cp or mv?

15.

Why do command names and filenames usually not have embedded SPACEs? How would you create a filename containing a SPACE? How would you remove it? (This is a thought exercise, not recommended practice. If you want to experiment, create and work in a directory that contains only your experimental file.)

16.

Create a file named answer and give the following command:

$ > answers.0102 < answers cat

Explain what the command does and why. What is a more conventional way of expressing this command?


Part II: The Editors

CHAPTER 6 THE vim EDITOR

CHAPTER 7 THE emacs EDITOR


Chapter 6. The vim Editor

Previous Page
Next Page

6. The vim Editor

IN THIS CHAPTER

Tutorial: Creating and Editing a File with vim | 147
Introduction to vim Features | 154
Online Help | 154
Command Mode: Moving the Cursor | 160
Input Mode | 163
Command Mode: Deleting and Changing Text | 165
Searching and Substituting | 168
Yank, Put, and Delete Commands | 175
The General-Purpose Buffer | 176
Reading and Writing Files | 178
The .vimrc Startup File | 180

This chapter begins with a history and description of vi, the original, powerful, sometimes cryptic, interactive, visually oriented text editor. The chapter continues with a tutorial that explains how to use vim (vi improveda vi clone supplied with Mac OS X 10.3 and later and available for most UNIX distributions) to create and edit a file. Much of the tutorial and the balance of the chapter apply to vi and other vi clones. Following the tutorial, the chapter delves into the details of many vim commands and explains how to use parameters to customize vim to meet your needs. It concludes with a quick reference/summary of vim commands. The vim home page is www.vim.org.

Mac OS X 10.2 and earlier used the nvi clone of vi. Not all of this chapter applies to nvi, although both vim and nvi are largely compatible with vi.


History

Before vi was developed, the standard UNIX system editor was ed (still available on Mac OS X systems), a line-oriented editor that made it difficult to see the context of your editing. Next came ex,[1] a superset of ed. The most notable advantage that ex has over ed is a display-editing facility that allows you to work with a full screen of text instead of just a line. While using ex, you can bring up the display-editing facility by giving a vi (Visual mode) command. People used this display-editing facility so extensively that the developers of ex made it possible to start the editor with the display-editing facility already running, without having to start ex and then give a vi command. Appropriately they named the program vi. You can call the Visual mode from ex, and you can go back to ex while you are using vi. Start by running ex; give a vi command to switch to Visual mode, and give a Q command while in Visual mode to use ex. Give a quit command to exit from ex.

[1] Under OS X the ex program is a link to vim.

vi clones

There are a number of versions, or clones, of vi. The most popular vi clones are elvis (elvis.the-little-red-haired-girl.org), nvi (an implementation of the original vi editor, www.bostic.com/vi), vile (dickey.his.com/vile/vile.html), and vim (www.vim.org). Each clone offers additional features beyond those provided with the original vi. The examples in this book are based on vim.

If you use one of the clones other than vim, or vi itself, you may notice slight differences from the examples presented in this chapter. The vim editor is compatible with almost all vi commands and runs on many platforms, including Windows, Macintosh, OS/2, UNIX, and Linux. Refer to the vim home page (www.vim.org) for more information and a very useful Tips section.

What vim is not

The vim editor is not a text formatting program. It does not justify margins or provide the output formatting features of a sophisticated word processing system such as AppleWorks. Rather, vim is a sophisticated text editor meant to be used to write code (C, HTML, Java, and so on), short notes, and input to a text formatting system, such as groff or TRoff. You can use fmt (page 736) to do minimal formatting on a text file that you create with vim.

Reading this chapter

Because vim is so large and powerful, this chapter describes only some of its features. Nonetheless, if vim is completely new to you, you may find even this limited set of commands overwhelming. The vim editor provides a variety of ways to accomplish any specified editing task. A useful strategy for learning vim is to begin by learning a subset of commands to accomplish basic editing tasks. Then, as you become more comfortable with the editor, you can learn other commands that enable you to do things more quickly and efficiently. The following tutorial section introduces a basic but useful set of vim commands and features that create and edit a file.


Tutorial: Creating and Editing a File with vim

This section explains how to start vim, enter text, move the cursor, correct text, save the file to the disk, and exit from vim. The tutorial discusses two of the modes of operation of vim and explains how to switch from one mode to the other.

vimtutor

In addition to working with this tutorial, you may want to try vim's tutor, named vimtutor: Give its name as a command to run it.

Specifying a terminal

Because vim takes advantage of features that are specific to various kinds of terminals, you must tell it what type of terminal or terminal emulator you are using. On many systems, and usually when you work on a terminal emulator such as Terminal, your terminal type is set automatically. If you need to specify your terminal type, refer to "Specifying a Terminal" on page 913.

Tip: The vi command runs vim

On some systems the command vi runs a minimal build of vim that is compact and faster to load than vim but includes fewer features. See "The compatible Parameter" on page 153 for information on running vim in vi compatible mode.

Starting vim

Start vim with the following command line to create and edit a file named practice:

$ vim practice

When you press RETURN, the command line disappears, and the screen looks similar to the one shown in Figure 6-1.

Figure 6-1. Starting vim

The tildes (~) at the left of the screen indicate that the file is empty. They disappear as you add lines of text to the file. If your screen looks like a distorted version of the one shown, your terminal type is probably not set correctly.

If you start vim with a terminal type that is not in the terminfo database, vim displays an error message and the terminal type defaults to ansi, which works on many terminals. In the following example, the user mistyped vt100 and set the terminal type to vg100:

Terminal entry not found in terminfo
'vg100' not known. Available builtin terminals are:
    builtin_ansi
    builtin_xterm
    builtin_iris-ansi
    builtin_dumb
defaulting to 'ansi'

If you want to reset the terminal type, press ESCAPE and then give the following command to exit from vim and get the shell prompt back:

:q!

When you enter the colon (:), vim moves the cursor to the bottom line of the screen. The characters q! tell vim to quit without saving your work. (You will not ordinarily exit from vim this way because you typically want to save your work.) You must press RETURN after you give this command. Once you get the shell prompt back, refer to "Specifying a Terminal" on page 913, and then start vim again.

If you start it without a filename, vim assumes that you are a novice and tells you how to get started (Figure 6-2).

Figure 6-2. Starting vim without a filename

The practice file is new so it does not contain any text. The vim editor displays a message similar to the one shown in Figure 6-1 on the status (bottom) line of the terminal to show that you are creating and editing a new file. When you edit an existing file, vim displays the first few lines of the file and gives status information about the file on the status line.

Command and Input Modes

Two of vim's modes of operation are Command mode (also called Normal mode) and Input mode (Figure 6-3). While vim is in Command mode, you can give vim commands. For example, you can delete text or exit from vim. You can also command vim to enter Input mode. In Input mode, vim accepts anything you enter as text and displays it on the screen. Press ESCAPE to return vim to Command mode.

Figure 6-3. Modes in vim

By default the vim editor keeps you informed about which mode it is in. You will see - - INSERT - - at the lower-left corner of the screen while vim is in Insert mode.

The following command causes vim to display line numbers next to the text you are editing:

:set number RETURN

Last Line mode

The colon (:) in the preceding command puts vim into another mode, Last Line mode. While in this mode, vim keeps the cursor on the bottom line of the screen. When you finish entering the command by pressing RETURN, vim restores the cursor to its place in the text. Give the command :set nonumber RETURN to turn off line numbers.

vim is case sensitive

When you give vim a command, remember that the editor is case sensitive. Thus vim editor interprets the same letter as two different commands, depending on whether you enter an uppercase or lowercase character. Beware of the CAPS LOCK (SHIFTLOCK) key. If you set this key to enter uppercase text while you are in Input mode and then exit to Command mode, vim interprets your commands as uppercase letters. It can be confusing when this happens because vim does not appear to be executing the commands you are entering.

Entering Text

Input mode (i/a)

When you start vim, you must put it in Input mode before you can enter text. To put vim in Input mode, press the i key (insert before the cursor) or the a key (append after the cursor).

If you are not sure whether vim is currently in Input mode, press the ESCAPE key; vim returns to Command mode if it was in Input mode or beeps, flashes, or does nothing if it is already in Command mode. You can put vim back in Input mode by pressing the i or a key again.

While vim is in Input mode, you can enter text by typing on the keyboard. If the text does not appear on the screen as you type, you are not in Input mode.

To continue with this tutorial, enter the sample paragraph shown in Figure 6-4, pressing the RETURN key to end each line. If you do not press RETURN before the cursor reaches the right side of the screen or window, vim will wrap the text so that it appears to start a new line. Physical lines will not correspond to programmatic (logical) lines in this situation, and editing will be more difficult.

Figure 6-4. Entering text with vim

While you are using vim, you can always correct any typing mistakes you make. If you notice a mistake on the line you are entering, you can correct it before you continue (page 151). You can correct other mistakes later. When you finish entering the paragraph, press ESCAPE to return vim to Command mode.

Getting Help

To get help while you are using vim, give the command :help [feature] followed by RETURN (you must be in Command mode when you give this command). The colon puts the cursor on the last line of the screen. If you type :help, vim displays an introduction to vim Help (Figure 6-5). Each dark band near the bottom of the screen names the file that is displayed above it. (Each area of the screen that displays a file, such as the two areas shown in Figure 6-5, is a vim "window.") The help.txt file occupies most of the screen (the upper window) in Figure 6-5. The file that is being edited (practice) occupies a few lines in the lower portion of the screen (the lower window).

Figure 6-5. The main vim Help screen

Read through the introduction to Help by scrolling the text as you read. Pressing j or the DOWN ARROW key moves the cursor down one line at a time; pressing CONTROL-D or CONTROL-U scrolls the cursor down or up half a window at a time. Give the command :q! to close the Help window.

You can get help with the insert commands by giving the command :help insert while vim is in Command mode (Figure 6-6).

Figure 6-6. Help with insert

Correcting Text as You Insert It

The keys that back up and correct a shell command line serve the same functions when vim is in Input mode. These keys include the erase, line kill, and word kill keys (usually CONTROL-H, CONTROL-U, and CONTROL-W, respectively). Although vim may not remove deleted text from the screen as you back up over it, the editor does remove it when you type over it or press RETURN.

Moving the Cursor

You need to be able to move the cursor on the screen so that you can delete, insert, and correct text. While vim is in Command mode, you can use the RETURN key, the SPACE bar, and the ARROW keys to move the cursor. If you prefer to keep your hand closer to the center of the keyboard, if your terminal does not have ARROW keys, or if the emulator you are using does not support them, you can use the h, j, k, and l (lowercase "l") keys to move the cursor left, down, up, and right, respectively.

Deleting Text

Delete character (x) Delete word (dw) Delete line (dd)

You can delete a single character by moving the cursor until it is over the character you want to delete and then giving the command x. You can delete a word by positioning the cursor on the first letter of the word and then giving the command dw (Delete word). You can delete a line of text by moving the cursor until it is anywhere on the line and then giving the command dd.

Undoing Mistakes

Undo (u)

If you delete a character, line, or word by mistake or give any command you want to undo, give the command u (Undo) immediately after the command you want to undo. The vim editor will restore the text to the way it was before you gave the last command. If you give the u command again, vim will undo the command you gave before the one it just undid. You can use this technique to back up over many of your actions. With the compatible parameter (page 153) set, vim can undo only the most recent change.

Redo (:redo)

If you undo a command you did not mean to undo, give a Redo command: CONTROL-R or :redo (followed by a RETURN). The vim editor will redo the undone command. As with the Undo command, you can give the Redo command many times in a row.

Entering Additional Text

Insert (i) Append (a)

When you want to insert new text within existing text, move the cursor so it is on the character that follows the new text you plan to enter. Then give the i (Insert) command to put vim in Input mode, enter the new text, and press ESCAPE to return vim to Command mode. Alternatively, you can position the cursor on the character that precedes the new text, and use the a (Append) command.

Open (o and O)

To enter one or more lines, position the cursor on the line above where you want the new text to go. Give the command o (Open). The vim editor opens a blank line, puts the cursor on it, and goes into Input mode. Enter the new text, ending each line with a RETURN. When you are finished entering text, press ESCAPE to return vim to Command mode. The O command works in the same way o works, except that it opens a line above the line the cursor is on.

Correcting Text

To correct text, use dd, dw, or x to remove the incorrect text. Then use i, a, o, or O to insert the correct text.

For example, to change the word pressing to hitting in Figure 6-4 on page 150 you might use the ARROW keys to move the cursor until it is on top of the p in pressing. Then give the command dw to delete the word pressing. Put vim in Input mode by giving an i command, enter the word hitting followed by a SPACE, and press ESCAPE. The word is changed and vim is in Command mode, waiting for another command. A shorthand for the two commands dw followed by the i command is cw (Change word). The command cw puts vim into Input mode.

Tip: Page breaks for the printer

CONTROL-L is a signal to a printer to skip to the top of the next page. You can enter this character anywhere in a document by pressing CONTROL-L while you are in Input mode. If ^L does not appear, press CONTROL-V before CONTROL-L.

Ending the Editing Session

While you are editing, vim keeps the edited text in an area named the Work buffer. When you finish editing, you must write out the contents of the Work buffer to a disk file so that the edited text is saved and available when you next want it.

Make sure that vim is in Command mode, and then use the ZZ command (you must use uppercase Z's) to write your newly entered text to the disk and end the editing session. After you give the ZZ command, vim returns control to the shell. You can exit with :q! if you do not want to save your work. Refer to page 191 for a summary of vim commands.


The compatible Parameter

The compatible parameter makes vim more compatible with vi. By default this parameter is set so that vim works like vi. If you set up a ~/.vimrc startup file (page 180), the compatible parameter is unset and vim works like vim. To get started with vim you can ignore this parameter.

Caution: Do not confuse ZZ with CONTROL-Z

When you exit from vim with ZZ, make sure that you type ZZ and not CONTROL-Z (typically the suspend key). When you press CONTROL-Z, vim disappears from your screen, almost as though you had exited from it. In fact, vim will continue running in the background with your work unsaved. Refer to "Job Control" on page 272. If you try to start editing the same file with a new vim command, vim displays a message about a swap file; refer to "File Locks" on page 157.

Setting the compatible parameter changes many aspects of how vim works. For example, when the compatible parameter is set, the Undo command (page 152) can undo only your most recent change; with the compatible parameter unset, you can call undo repeatedly to undo many changes. This chapter notes when the compatible parameter affects a command. To obtain more details on the compatible parameter, give the command :help compatible RETURN. For a complete list of how vim differs from the original vi, use :help vi-diff RETURN. See page 151 for a discussion of the help command.

From the command line use the C option to set the compatible parameter and the N option to unset it. Refer to "Setting Parameters from Within vim" on page 179 for information on how to change the compatible parameter while you are running vim.


Introduction to vim Features

This section covers modes of operation, online help, the Work buffer, emergency procedures, and other vim features. To see which features are incorporated in a particular build, give a vim command followed by the version option.

Online Help

As covered briefly earlier, vim provides help while you are using it. Give the command :help feature to display information about feature. As you scroll through the various help texts you will see words with a bar on either side, such as |tutor|. These words are active links: Move the cursor on top of an active link and press CONTROL-] to jump to the linked text. Use CONTROL-O (lowercase "o") to jump back to where you were in the help text. You can also use the active link words in place of feature. For example, you might see the reference |credits|; you could enter :help credits RETURN to read more about credits. Enter :q! to close a help window.

Some common features that you may want to look up using the help system are insert, delete, and opening-window. Although opening-window is not intuitive, you will get to know the names of features as you spend more time with vim. You can also give the command :help doc-file-list to view a complete list of the help files. Although vim is a free program, the author requests that you donate the money you would have spent on similar software to help the kids in Uganda (:help uganda for more information).

Modes of Operation

The vim editor is part of the ex editor, which has five modes of operation:

  • ex Command mode

  • ex Input mode

  • vim Command mode

  • vim Input mode

  • vim Last Line mode

While in Command mode, vim accepts keystrokes as commands, responding to each command as you enter it. It does not display the characters you type in this mode. While in Input mode, vim accepts and displays keystrokes as text that it eventually puts into the file you are editing. All commands that start with a colon (:) put vim in Last Line mode. The colon moves the cursor to the bottom line of the screen, where you enter the rest of the command.

In addition to the position of the cursor, there is another important difference between Last Line mode and Command mode. When you give a command in Command mode, you do not terminate the command with a RETURN. However, you must terminate all Last Line mode commands with a RETURN.

You do not normally use the ex modes. When this chapter refers to Input and Command modes, it means the vim modes, not the ex modes.

At the start of an editing session, vim is in Command mode. Several commands, including Insert and Append, put vim in Input mode. When you press the ESCAPE key, vim always reverts to Command mode.

The Change and Replace commands combine the Command and Input modes. The Change command deletes the text you want to change and puts vim in Input mode so you can insert new text. The Replace command deletes the character(s) you overwrite and inserts the new one(s) you enter. Figure 6-3 on page 149 shows the modes and the methods for changing between them.

Tip: Watch the mode and the CAPS LOCK key

Almost anything you type in Command mode means something to vim. If you think that vim is in Input mode when it is actually in Command mode, typing in text can produce confusing results. When learning vim, make sure that the showmode parameter (page 184) is set (it is by default) to remind you which mode you are using. You may also find it useful to turn on the status line by giving a :set laststatus=2 command (page 182).

Also keep your eye on the CAPS LOCK key. In Command mode typing uppercase letters produces different results than typing lowercase ones. It can be disorienting to give commands and have vim give the "wrong" responses.

The Display

The vim editor uses the status line and several special symbols to give information about what is happening during an editing session.

Status Line

The vim editor displays status information on the bottom line of the display area. This information includes error messages, information about the deletion or addition of blocks of text, and file status information. In addition, vim displays Last Line mode commands on the status line.

Redrawing the Screen

Sometimes the screen becomes garbled or overwritten. When vim puts characters on the screen, it sometimes leaves @ on a line instead of deleting the line. When output from a program becomes intermixed with the display of the Work buffer things can get confusing. The output does not become part of the Work buffer but affects only the display. If the screen gets overwritten, press ESCAPE to make sure vim is in Command mode, and press CONTROL-L to redraw (refresh) the screen.

Tilde (~) Symbol

If the end of the file is displayed on the screen, vim marks lines that would appear past the end of the file with a tilde (~) at the left of the screen. When you start editing a new file, the vim editor marks each line on the screen (except for the first line) with this symbol.

Correcting Text as You Insert It

While vim is in Input mode, you can use the erase and line kill keys to back up over text so you can correct it. You can also use CONTROL-W to back up over words.

Work Buffer

The vim editor does all of its work in the Work buffer. At the start of an editing session, vim reads the file you are editing from the disk into the Work buffer. During the editing session, it makes all changes to this copy of the file but does not change the file on the disk until you write the contents of the Work buffer back to the disk. Normally when you end an editing session, you command vim to write out the contents of the Work buffer, which makes the changes to the text final. When you edit a new file, vim creates the file when it writes the contents of the Work buffer to the disk, usually at the end of the editing session.

Storing the text you are editing in the Work buffer has both advantages and disadvantages. If you accidentally end an editing session without writing out the contents of the Work buffer, your work is lost. However, if you unintentionally make some major changes (such as deleting the entire contents of the Work buffer), you can end the editing session without implementing the changes.

If you want to use vim to look at a file but not to change it, you can use the view utility:

$ view filename

Calling the view utility is the same as calling the vim editor with the R (readonly) option. Once you have invoked the editor in this way, you cannot write the contents of the Work buffer back to the file whose name appeared on the command line. You can always write the Work buffer out to a file with a different name.

Line Length and File Size

The vim editor operates on any format file, provided the length of a single "line" (that is, the characters between two NEWLINE characters) can fit into available memory. The total length of the file is limited only by available disk space and memory.

Windows

The vim editor allows you to open, close, and hide multiple windows, each of which allows you to edit a different file. Most of the window commands consist of CONTROL-W followed by another letter. For example, CONTROL-W s opens another window (splits the screen) that is editing the same file. CONTROL-W n opens a second window that is editing an empty file. CONTROL-W w moves the cursor between windows, and CONTROL-W q (or :q) quits (closes) a window. Give the command :help windows to display a complete list of windows commands.

File Locks

When you edit an existing file, vim displays the first few lines of the file, gives status information about the file on the status line, and locks the file. When you try to open a locked file with vim, you will see a message similar to the one shown in Figure 6-7. You will see this type of message in two cases: when you try to edit a file that someone is already editing (perhaps you are editing it in another window or on another terminal) or when you try to edit a file that you were editing when vim or the system crashed.

Figure 6-7. Attempting to open a locked file

Although it is advisable to follow the instructions that vim displays, a second user can edit a file and write it out with a different filename. Refer to the next sections for more information.

Abnormal Termination of an Editing Session

You can end an editing session in one of two ways: When you exit from vim, you can save the changes you made during the editing session or you can abandon those changes. You can use the ZZ or :wq command from Command mode to save your changes and exit from vim (page 153).

To end an editing session without writing out the contents of the Work buffer, give the following command:

:q!

When you use this command to end an editing session, vim does not preserve the contents of the Work buffer, so you will lose all the work you did since the last time you wrote the Work buffer to disk. The next time you edit or use the file, it will appear as it did the last time you wrote the Work buffer to disk. Use the :q! command cautiously.

Sometimes you may find that you created or edited a file but vim will not let you exit. For example, if you forgot to specify a filename when you first called vim, you will get a message saying No file name when you give a ZZ command. If vim does not let you exit normally, you can use the Write command (:w) to name the file and write it to disk before you quit vim. Give the following command, substituting the name of the file for filename (remember to follow the command with a RETURN):

:w filename

After you give the Write command, you can use :q to quit using vim. You do not need to use the exclamation point (as in q!); it is necessary only when you have made changes since the last time you wrote the Work buffer to disk. Refer to page 178 for more information about the Write command.

Tip: When you cannot write to a file

It may be necessary to write a file using :w filename if you do not have write permission for the file you are editing. If you give a ZZ command and see the message "filename" is read only, you do not have write permission for the file. Use the Write command with a temporary filename to write the file to disk under a different filename. If you do not have write permission for the working directory, vim may still not be able to write the file to the disk. Give the command again, using an absolute pathname of a dummy (nonexistent) file in your home directory in place of the filename. (For example, Alex might give the command :w /Users/alex/temp or :w ~/temp.)

If vim reports File exists, you will need to use :w! filename to overwrite the existing file (make sure that you want to do this). Refer to page 179.

Recovering Text After a Crash

The vim editor temporarily stores the file you are working on in a swap file. If the system crashes while you are editing a file with vim, you can often recover its text from the swap file. When you attempt to edit a file that has a swap file, you will see a message similar to the one shown in Figure 6-7 on page 157. If someone else is editing the file, quit or open the file as a readonly file.

Alex checks whether the swap file exists for a file named memo, which he was editing when the system went down:

$ vim -r
Swap files found:
   In current directory:
1.    .memo.swp
             dated: Mon Oct 18 13:16:06 2004
          owned by: alex
         file name: ~alex/memo
         host name: bravo.example.com
         user name: alex
        process ID: 19786
   In directory ~/tmp:
      -- none --
   In directory /var/tmp:
      -- none --
   In directory /tmp:
      -- none --

With the r option vim displays a list of any swap files that it has saved (some may be old). If your work was saved, give the same command followed by a SPACE and the name of the file. You will then be editing a recent copy of your Work buffer. Use :w filename immediately to save the salvaged copy of the Work buffer to disk under a name different from the original file. Then check the recovered file to make sure it is OK. Following is Alex's exchange with vim as he recovers memo. Subsequently he deletes the swap file:

$ vim -r memo
Using swap file ".memo.swp"
Original file "~/memo"
Recovery completed. You should check if everything is OK.
(You might want to write out this file under another name
and run diff with the original file to check for changes)
Delete the .swp file afterwards.

Hit ENTER or type command to continue
:w memo2
:q
$ rm .memo.swp

Tip: You must recover files on the system you were using

The recovery feature of vim is specific to the system you were using when the crash occurred. If you are running on a cluster, you must log in on the system you were using before the crash to use the r option.


Command Mode: Moving the Cursor

While vim is in Command mode, you can position the cursor over any character on the screen. You can also display a different portion of the Work buffer on the screen. By manipulating the screen and cursor position, you can place the cursor on any character in the Work buffer.

You can move the cursor forward or backward through the text. As illustrated in Figure 6-8, forward means toward the right and bottom of the screen and the end of the file. Backward means toward the left and top of the screen and the beginning of the file. When you use a command that moves the cursor forward past the end (right) of a line, the cursor generally moves to the beginning (left) of the next line. When you move it backward past the beginning of a line, the cursor moves to the end of the previous line.

Figure 6-8. Forward and backward

Long lines

Sometimes a line in the Work buffer may be too long to appear as a single line on the screen. In such a case vim wraps the current line onto the next line (unless you set the nowrap option [page 182]).

You can move the cursor through the text by any Unit of Measure (that is, character, word, line, sentence, paragraph, or screen). If you precede a cursor-movement command with a number, called a Repeat Factor, the cursor moves that number of units through the text. Refer to pages 188 and 191 for more precise definitions of these terms.

Moving the Cursor by Characters

l/h

The SPACE bar moves the cursor forward, one character at a time, toward the right side of the screen. The l (lowercase "l") key and the RIGHT ARROW key (Figure 6-9) do the same thing. The command 7 SPACE or 7l moves the cursor seven characters to the right. These keys cannot move the cursor past the end of the current line to the beginning of the next line. The h and LEFT ARROW keys are similar to the l key but work in the opposite direction.

Figure 6-9. Moving the cursor by characters

Moving the Cursor to a Specific Character

f/F

You can move the cursor to the next occurrence of a specified character on the current line by using the Find command. For example, the following command moves the cursor from its current position to the next occurrence of the character a, if one appears on the same line:

fa

You can also find the previous occurrence by using a capital F. The following command moves the cursor to the position of the closest previous a in the current line:

Fa

A semicolon (;) repeats the last Find command.

Moving the Cursor by Words

w/W

The w (word) key moves the cursor forward to the first letter of the next word (Figure 6-10). Groups of punctuation count as words. This command goes to the next line if the next word is located there. The command 15w moves the cursor to the first character of the fifteenth subsequent word.

Figure 6-10. Moving the cursor by words

The W key is similar to the w key but moves the cursor by blank-delimited words, including punctuation, as it skips forward. (Refer to "Blank-Delimited Word" on page 188.)

b/B

The b (back) key moves the cursor backward to the first letter of the previous word. The B key moves the cursor backward by blank-delimited words. Similarly the e key moves the cursor to the end of the next word; E moves it to the end of the next blank-delimited word.

Moving the Cursor by Lines

j/k

The RETURN key moves the cursor to the beginning of the next line; the j and DOWN ARROW keys move it down one line to the character just below the current character (Figure 6-11). If no character is immediately below the current character, the cursor moves to the end of the next line. The cursor will not move past the last line of text in the work buffer.

Figure 6-11. Moving the cursor by lines

The k and UP ARROW keys are similar to the j key but work in the opposite direction. The minus () key is similar to the RETURN key but works in the opposite direction.

Moving the Cursor by Sentences and Paragraphs

)/( }/{

The ) and } keys move the cursor forward to the beginning of the next sentence or the next paragraph, respectively (Figure 6-12). The ( and { keys move the cursor backward to the beginning of the current sentence or paragraph. You can find more information on sentences and paragraphs starting on page 189.

Figure 6-12. Moving the cursor by sentences, paragraphs, H, M, and L

[View full size image]

Moving the Cursor Within the Screen

H/M/L

The H (home) key positions the cursor at the left end of the top line of the screen. The M (middle) key moves the cursor to the middle line, and the L (lower) key moves it to the bottom line (Figure 6-12).

Viewing Different Parts of the Work Buffer

The screen displays a portion of the text that is in the Work buffer. You can display the text preceding or following the text on the screen by scrolling the display. You can also display a portion of the Work buffer based on a line number.

CONTROL-D CONTROL-U

Press CONTROL-D to scroll the screen down (forward) through the file so that vim displays half a screen of new text. Use CONTROL-U to scroll the screen up (backward) by the same amount. If you precede either of these commands with a number, vim will scroll that number of lines each time you use CONTROL-D or CONTROL-U for the rest of the session (unless you again change the number of lines to scroll). See page 183 for a discussion of the scroll parameter.

CONTROL-F CONTROL-B

The CONTROL-F (forward) and CONTROL-B (backward) keys display almost a whole screen of new text, leaving a couple of lines from the previous screen for continuity. On many keyboards you can use the PAGE DOWN and PAGE UP keys in place of CONTROL-F and CONTROL-B.

Line numbers (G)

When you enter a line number followed by G (goto), vim positions the cursor on that line in the Work buffer. If you press G without a number, vim positions the cursor on the last line in the Work buffer. Line numbers are implicit; your file does not need to have actual line numbers for this command to work. Refer to "Line numbers" on page 182 if you want vim to display line numbers.


Input Mode

The Insert, Append, Open, Change, and Replace commands put vim in Input mode. While vim is in this mode, you can put new text into the Work buffer. You need to press the ESCAPE key to return vim to Command mode when you finish entering text. Refer to "Show mode" on page 184 if you want vim to remind you when it is in Input mode (it does by default).

Inserting Text

Insert (i/I)

The i (Insert) command puts vim in Input mode and places the text you enter before the character the cursor is on (the current character). The I command places text at the beginning of the current line (Figure 6-13). Although the i and I commands sometimes overwrite text on the screen, the characters in the Work buffer are not changed; only the display is affected. The overwritten text is redisplayed when you press ESCAPE and vim returns to Command mode. Use i or I to insert a few characters or words into existing text or to insert text in a new file.

Figure 6-13. The I, i, a, and A commands

Appending Text

Append (a/A)

The a (Append) command is similar to the i command, except that it places the text you enter after the current character (Figure 6-13). The A command places the text after the last character on the current line.

Opening a Line for Text

Open (o/O)

The o (Open) and O commands open a blank line within existing text, place the cursor at the beginning of the new (blank) line, and put vim in Input mode. The O command opens a line above the current line; o opens one below the current line. Use the Open commands when you are entering several new lines within existing text.

Replacing Text

Replace (r/R)

The r and R (Replace) commands cause the new text you enter to overwrite (replace) existing text. The single character you enter following an r command overwrites the current character. After you enter that character, vim automatically returns to Command modeyou do not need to press the ESCAPE key.

The R command causes all subsequent characters to overwrite existing text until you press ESCAPE to return vim to Command mode.

Tip: Replacing TABs

The Replace commands may appear to behave strangely when you replace TAB characters. TAB characters can appear as several SPACEsuntil you try to replace them. They are actually only one character and are replaced by a single character. Refer to "Invisible characters" on page 182 for information on how to display TABs as visible characters.

Quoting Special Characters in Input Mode

CONTROL-V

While you are in Input mode, you can use the Quote command, CONTROL-V, to enter any character into the text, including characters that normally have special meaning to vim. Among these characters are CONTROL-L (or CONTROL-R), which redraws the screen; CONTROL-W, which backs the cursor up a word to the left; and ESCAPE, which ends Input mode.

To insert one of these characters into the text, type CONTROL-V followed by the character. CONTROL-V quotes the single character that follows it. For example, to insert the sequence ESCAPE[2J into a file you are creating in vim, you type the character sequence CONTROL-V ESCAPE[2J. This character sequence clears the screen of a DEC VT-100 and other similar terminals. Although you would not ordinarily want to type this sequence into a document, you might want to use it or another ESCAPE sequence in a shell script you are creating in vim. Refer to Chapter 13 for information about writing shell scripts.


Command Mode: Deleting and Changing Text

This section describes the commands to delete and replace, or change, text in the document you are editing. The Undo command is covered here because it allows you to restore deleted or changed text.

Undoing Changes

Undo (u/U)

The u command (Undo) restores text that you just deleted or changed by mistake. A single Undo command restores only the most recently deleted text. If you delete a line and then change a word, the first Undo restores only the changed word; you have to give a second Undo command to restore the deleted line. The U command restores the last line you changed to the way it was before you started changing it, even after several changes.

Deleting Characters

Delete character (x/X)

The x command deletes the current character. You can precede the x command by a Repeat Factor (page 191) to delete several characters on the current line, starting with the current character. The X command deletes characters to the left of the cursor.

Deleting Text

Delete (d/D)

The d (Delete) command removes text from the Work buffer. The amount of text that d removes depends on the Repeat Factor and the Unit of Measure (page 188) you enter after the d. After the text is deleted, vim is still in Command mode.

Tip: Use dd to delete a single line

The command d RETURN deletes two lines: the current line and the following one. Use dd to delete just the current line, or precede dd by a Repeat Factor (page 191) to delete several lines.

You can delete from the current cursor position up to a specific character on the same line. To delete up to the next semicolon (;), give the command dt; (see page 169 for more information on the t command). To delete the remainder of the current line, use D or d$. Table 6-1 lists some Delete commands. Each command, except the last group that starts with dd, deletes from/to the current character.

Table 6-1. Delete command examples

Command

Result

dl

Deletes current character (same as the x command)

d0

Deletes from beginning of line

d^

Deletes from first character of the line (not including leading SPACEs or TABs)

dw

Deletes to end of word

d3w

Deletes to end of third word

db

Deletes from beginning of word

dW

Deletes to end of blank-delimited word

dB

Deletes from beginning of blank-delimited word

d7B

Deletes from seventh previous beginning of blank-delimited word

d)

Deletes to end of sentence

d4)

Deletes to end of fourth sentence

d(

Deletes from beginning of sentence

d}

Deletes to end of paragraph

d{

Deletes from beginning of paragraph

d7{

Deletes from seventh paragraph preceding beginning of paragraph

d/text

Deletes up to the next occurrence of word text

dfc

Deletes on current line up to and including next occurrence of character c

dtc

Deletes on current line up to next occurrence of c

D

Deletes to end of line

d$

Deletes to end of line

dd

Deletes the current line

5dd

Deletes five lines starting with the current line

dL

Deletes through last line on screen

dH

Deletes from first line on screen

dG

Deletes through end of Work buffer

d1G

Deletes from beginning of Work buffer

Tip: Exchange characters and lines

If two characters are out of order, position the cursor on the first character and give the commands xp. If two lines are out of order, position the cursor on the first line and give the commands ddp. See page 176 for more information on the Put commands.

Changing Text

Change (c/C)

The c (Change) command replaces existing text with new text. The new text does not have to occupy the same space as the existing text. You can change a word to several words, a line to several lines, or a paragraph to a single character. The C command replaces the text from the cursor position to the end of the line.

The c command deletes the amount of text specified by the Repeat Factor and the Unit of Measure (page 188) and puts vim in Input mode. When you finish entering the new text and press ESCAPE, the old word, line, sentence, or paragraph is changed to the new one. Pressing ESCAPE without entering new text deletes the specified text (replaces the specified text with nothing).

Table 6-2 lists some Change commands. Except for the last two, each command changes text from/to the current character.

Table 6-2. Change command examples

Command

Result

cl

Changes current character

cw

Changes to end of word

c3w

Changes to end of third word

cb

Changes from beginning of word

cW

Changes to end of blank-delimited word

cB

Changes from beginning of blank-delimited word

c7B

Changes from beginning of seventh previous blank-delimited word

c$

Changes to end of line

c0

Changes from beginning of line

c)

Changes to end of sentence

c4)

Changes to end of fourth sentence

c(

Changes from beginning of sentence

c}

Changes to end of paragraph

c{

Changes from beginning of paragraph

c7{

Changes from beginning of seventh preceding paragraph

ctc

Changes on current line up to next occurrence of c

C

Changes to end of line

cc

Changes the current line

5cc

Changes five lines starting with the current line

Tip: dw works differently from cw

The dw command deletes all the characters through (including) the SPACE at the end of a word. The cw command changes only the characters in the word, leaving the trailing SPACE intact.

Replacing Text

Substitute (s/S)

The s and S (Substitute) commands also replace existing text with new text (Table 6-3). The s command deletes the character the cursor is on and puts vim into Input mode. It has the effect of replacing the single character that the cursor is on with whatever you type until you press ESCAPE. The S command does the same thing as the cc command: It changes the current line. The s command replaces characters only on the current line. If you specify a Repeat Factor before an s command and this action would replace more characters than exist on the current line, s changes characters only to the end of the line (same as C).

Table 6-3. Substitute command examples

Command

Result

s

Substitutes one or more characters for current character

S

Substitutes one or more characters for current line

5s

Substitutes one or more characters for five characters, starting with current character

Changing Case

The tilde (~) character changes the case of the character under the cursor from uppercase to lowercase, or vice versa. You can precede the tilde with a number to specify the number of characters you want the command to affect. For example, 5~ will transpose the next five characters starting with the character under the cursor, but it will not transpose characters past the end of the line the cursor is on.


Searching and Substituting

Searching for and replacing a character, a string of text, or a string that is matched by a regular expression is a key feature of any editor. The vim editor provides simple commands for searching for a character on the current line. It also provides more complex commands for searching for and optionally substituting for single and multiple occurrences of strings or regular expressions anywhere in the Work buffer.

Searching for a Character

Find (f/F)

You can search for and move the cursor to the next occurrence of a specified character on the current line using the f (Find) command. Refer to "Moving the Cursor to a Specific Character" on page 161.

t/T

The next two commands are used in the same manner as the Find command. The t command places the cursor on the character before the next occurrence of the specified character. The T command places the cursor on the character after the previous occurrence of the specified character.

A semicolon (;) repeats the last f, F, t, or T command.

You can combine these search commands with other commands. For example, the command d2fq deletes the text from the location of the cursor to the second occurrence of the letter q on the current line.

Searching for a String

The vim editor can search backward or forward through the Work buffer to find a string of text or a string that matches a regular expression (see Appendix A). To find the next occurrence of a string (forward), press the forward slash (/) key, enter the text you want to find (called the search string), and press RETURN. When you press the slash key, vim displays a slash on the status line. As you enter the string of text, it is also displayed on the status line. When you press RETURN, vim searches for the string. If this search is successful, vim positions the cursor on the first character of the string. If you use a question mark (?) in place of the forward slash, vim searches for the previous occurrence of the string. If you need to include a forward slash in a forward search or a question mark in a backward search, you must quote it by preceding it with a backslash (\).

Tip: Two distinct ways of quoting characters

You use CONTROL-V to quote special characters in text that you are entering into a file (page 164). This section discusses the use of a backslash (\) to quote special characters in a search string. The two techniques of quoting characters are not interchangeable.

The N and n keys repeat the last search without any need for you to reenter the search string. The n key repeats the original search exactly, and the N key repeats the search in the opposite direction of the original search.

If you are searching forward and vim does not find the search string before it gets to the end of the Work buffer, the editor typically wraps around and continues the search at the beginning of the Work buffer. During a backward search, vim wraps around from the beginning of the Work buffer to the end. Also, vim normally performs case-sensitive searches. Refer to "Wrap scan" (page 184) and "Ignore case in searches" (page 182) for information about how to change these search parameters.

Normal Versus Incremental Searches

When vim performs a normal search (its default behavior), you enter a slash or question mark followed by the search string and press RETURN. The vim editor moves the cursor to the next or previous occurrence of the string you are searching for.

When vim performs an incremental search, you enter a slash or question mark. As you enter each character of the search string, vim moves the highlight to the next or previous occurrence of the string you have entered so far. When the highlight is on the string you are searching for, you must press RETURN to move the cursor to the highlighted string. If the string you enter does not match any text, vim does not highlight anything.

The type of search that vim performs depends on the incsearch parameter (page 182). Give the command :set incsearch to turn on incremental searching. Use noincsearch to turn it off. When you set the compatible parameter (page 153), vim turns off incremental searching.

Special Characters in Search Strings

Because the search string is a regular expression, some characters take on a special meaning within the search string. The following paragraphs list some of these characters. See also "Extended Regular Expressions" on page 903.

The first two items in the following list (^ and $) always have their special meanings within a search string unless you quote them by preceding them with a backslash (\). You can turn off the special meanings within a search string for the rest of the items in the list by setting the nomagic parameter. See "Allow special characters in searches" (page 181) for more information.

^ Beginning-of-Line Indicator

When the first character in a search string is a caret (also called a circumflex) it matches the beginning of a line. For example, the command /^the finds the next line that begins with the string the.

$ End-of-Line Indicator

A dollar sign matches the end of a line. For example, the command /!$ finds the next line that ends with an exclamation point and / $ matches the next line that ends with a SPACE.

. Any-Character Indicator

A period matches any character, anywhere in the search string. For example, the command /l..e finds line, followed, like, included, all memory, or any other word or character string that contains an l followed by any two characters and an e. To search for a period, use a backslash to quote the period (\.).

\> End-of-Word Indicator

This pair of characters matches the end of a word. For example, the command /s\> finds the next word that ends with an s. Whereas a backslash (\) is typically used to turn off the special meaning of a character, the character sequence \> has a special meaning, while > alone does not.

\< Beginning-of-Word Indicator

This pair of characters matches the beginning of a word. For example, the command /\<The finds the next word that begins with the string The. The beginning-of-word indicator uses the backslash in the same, atypical way as the end-of-word indicator.

* Zero or More Occurrences

This character is a modifier that will match zero or more occurrences of the character immediately preceding it. For example, the command /dis*m will match the string di followed by zero or more s characters followed by an m. Examples of successful matches are dim or dism or dissm.

[ ] Character-Class Definition

Brackets surrounding two or more characters match any single character located between the brackets. For example, the command /dis[ck] finds the next occurrence of either disk or disc.

There are two special characters you can use within a character-class definition. A caret (^) as the first character following the left bracket defines the character class to be any except the following characters. A hyphen between two characters indicates a range of characters. Refer to the examples in Table 6-4.

Table 6-4. Search examples

Search string

What it finds

/and

Finds the next occurrence of the string and

Examples: sand and standard slander andiron

/\<and\>

Finds the next occurrence of the word and

Example: and

/^The

Finds the next line that starts with The

Examples:

The ...

There ...

/^[0-9][0-9])

Finds the next line that starts with a two-digit number followed by a right parenthesis

Examples:

77)...

01)...

15)...

/\<[adr]

Finds the next word that starts with an a, d, or r

Examples: apple drive road argument right

/^[A-Za-z]

Finds the next line that starts with an uppercase or lowercase letter

Examples:

will not find a line starting with the number 7 ...

Dear Mr. Jones ...

in the middle of a sentence like this ...

Substituting One String for Another

A Substitute command combines the effects of a Search command and a Change command. That is, it searches for a string (regular expression) just as the / command does, allowing the same special characters discussed in the previous section. When it finds the string or matches the regular expression, the Substitute command changes the string or regular expression it matches. The syntax of the Substitute command is

:[g][address]s/search-string/replacement-string[/option]

As with all commands that begin with a colon, vim executes a Substitute command from the status line.

The Substitute Address

If you do not specify an address, Substitute searches only the current line. If you use a single line number as the address, Substitute searches that line. If the address is two line numbers separated by a comma, Substitute searches those lines and the lines between them. Refer to "Line numbers" on page 182 if you want vim to display line numbers. Wherever a line number is allowed in the address, you may also use an address-string enclosed between slashes. The vim editor operates on the next line that the address-string matches. When you precede the first slash of the address-string with the letter g (for global), vim operates on all lines in the file that the address-string matches. (This g is not the same as the one that goes at the end of the Substitute command to cause multiple replacements on a single line; see "Searching for and Replacing Strings" below).

Within the address, a period represents the current line, a dollar sign represents the last line in the Work buffer, and a percent sign represents the entire Work buffer. You can perform address arithmetic using plus and minus signs. Table 6-5 shows some examples of addresses.

Table 6-5. Addresses

Address

Portion of Work buffer addressed

5

Line 5

77,100

Lines 77 through 100 inclusive

1,.

Beginning of Work buffer through current line

.,$

Current line through end of Work buffer

1,$

Entire Work buffer

%

Entire Work buffer

/pine/

The next line containing the word pine

g/pine/

All lines containing the word pine

.,.+10

Current line through tenth following line (11 lines in all)

Searching for and Replacing Strings

An s comes after the address in the command syntax, indicating that this is a Substitute command. A delimiter follows the s, marking the beginning of the search-string. Although the examples in this book use a forward slash, you can use as a delimiter any character that is not a letter, number, blank, or backslash. You must use the same delimiter at the end of the search-string.

Next comes the search-string. It has the same format as the search string in the / command and can include the same special characters (page 170). (The search-string is a regular expression; refer to Appendix A for more information.) Another delimiter marks the end of the search-string and the beginning of the replace-string.

The replace-string replaces the text matched by the search-string. It should be followed by the delimiter character. You can omit the final delimiter when no option follows the replace-string; a final delimiter is required if an option is present.

Several characters have special meanings in the search-string, and other characters have special meanings in the replace-string. For example, an ampersand (&) in the replace-string represents the text that was matched by the search-string. A backslash in the replace-string quotes the character that follows it. Refer to Table 6-6 and Appendix A.

Table 6-6. Search and replace examples

Command

Result

:s/bigger/biggest/

Replaces the first occurrence of the string bigger on the current line with biggest

Example:

bigger [image] biggest

:1,.s/Ch 1/Ch 2/g

Replaces every occurrence of the string Ch 1, before or on the current line, with the string Ch 2

Examples:

Ch 1 [image] Ch 2

Ch 12 [image] Ch 22

:1,$s/ten/10/g

Replaces every occurrence of the string ten with the string 10

Examples:

ten [image] 10

often [image] of10

tenant [image] 10ant

:g/chapter/s/ten/10/

Replaces the first occurrence of the string ten with the string 10 on all lines containing the word chapter

Examples:

chapter ten [image] chapter 10

chapters will often [image] chapters will of10

:%s/\<ten\>/10/g

Replaces every occurrence of the word ten with the string 10

Example:

ten [image] 10

:.,.+10s/every/each/g

Replaces every occurrence of the string every with the string each on the current line through the tenth following line

Examples:

every [image] each

everything [image] eachthing

:s/\<short\>/"&"/

Replaces the word short on the current line with "short" (enclosed within quotation marks)

Example:

the shortest of the short [image] the shortest of the "short"

Normally, the Substitute command replaces only the first occurrence of any text that matches the search-string on a line. If you want a global substitutionthat is, if you want to replace all matching occurrences of text on a lineappend the g (global) option after the delimiter that ends the replace-string. Another useful option, c (check), causes vim to ask whether you would like to make the change each time it finds text that matches the search-string. Pressing y replaces the search-string, q terminates the command, l (last) makes the replacement and quits, a (all) makes all remaining replacements, and n continues the search without making that replacement.

The address-string need not be the same as the search-string. For example,

:/candle/s/wick/flame/

substitutes flame for the first occurrence of wick on the next line that contains the string candle. Similarly,

:g/candle/s/wick/flame/

performs the same substitution for the first occurrence of wick on each line of the file containing the string candle and

:g/candle/s/wick/flame/g

performs the same substitution for all occurrences of wick on each line that contains the string candle.

If the search-string is the same as the address, you can leave the search-string blank. For example, the command :/candle/s//lamp/ is equivalent to the command :/candle/s/candle/lamp/.


Miscellaneous Commands

This section describes three commands that do not fit naturally into any other groups.

Join

Join (J)

The J (Join) command joins the line below the current line to the end of the current line, inserting a SPACE between what was previously two lines and leaving the cursor on this SPACE. If the current line ends with a period, vim inserts two SPACEs.

You can always "unjoin" (break) a line into two lines by replacing the SPACE or SPACEs where you want to break the line with a RETURN.

Status

Status (CONTROL-G)

The Status command, CONTROL-G, displays the name of the file you are editing, information about whether the file has been modified, the total number of lines in the Work buffer, the percentage of the Work buffer preceding the current line, and the number of the line and character the cursor is on. You can also use :f to display status information. Following is a sample status line:

"memo.05" 17103 lines --3%--                         569,1          3%

. (Period)

The . (period) command repeats the most recent command that made a change. If you had just given a d2w command (delete the next two words), for example, the . command would delete the next two words. If you had just inserted text, the . command would repeat the insertion of the same text. This command is useful if you want to change some occurrences of a word or phrase in the Work buffer. Search for the first occurrence of the word (use /) and then make the change you want (use cw). You can then use n to search for the next occurrence of the word and . to make the same change to it. If you do not want to make the change, use n again to find the next occurrence.


Yank, Put, and Delete Commands

The vim editor has a General-Purpose buffer and 26 Named buffers that can hold text during an editing session. These buffers are useful if you want to move or copy a portion of text to another location in the Work buffer. A combination of the Delete and Put commands removes text from one location in the Work buffer and places it in another location in the Work buffer. The Yank and Put commands copy text to another location in the Work buffer, without changing the original text.

The General-Purpose Buffer

The vim editor stores the text that you most recently changed, deleted, or yanked in the General-Purpose buffer. The Undo command retrieves text from the General-Purpose buffer when it restores text.

Copying Text to the Buffer

Yank (y/Y)

The Yank command (y) is identical to the Delete (d) command except that it does not delete text from the Work buffer. The vim editor places a copy of the yanked text in the General-Purpose buffer. You can then use a Put command to place another copy of it elsewhere in the Work buffer. Use the Yank command just as you use the Delete command. The uppercase Y command yanks an entire line into the General-Purpose buffer.

Tip: Use yy to yank one line

Just as d RETURN deletes two lines, so y RETURN yanks two lines. Use the yy command to yank and dd to delete the current line.

Tip: D works differently from Y

The D command (page 165) does not work in the same manner as the Y command. Whereas D deletes to the end of the line, Y yanks the entire line regardless of the cursor position.

Copying Text from the Buffer

Put (p/P)

The Put commands, p and P, copy text from the General-Purpose buffer to the Work buffer. If you delete or yank characters or words into the General-Purpose buffer, p inserts them after the current character, and P inserts them before this character. If you delete or yank lines, sentences, or paragraphs, P inserts the contents of the General-Purpose buffer before the line the cursor is on, and p inserts them after this line.

Put commands do not destroy the contents of the General-Purpose buffer. Thus you can place the same text at several points within the file by using one Delete or Yank command and several Put commands.

Deleting Text Copies It into the Buffer

Any of the Delete commands described earlier in this chapter (page 165) place the deleted text in the General-Purpose buffer. Just as you can use the Undo command to put the deleted text back where it came from, so you can use a Put command to put the deleted text at another location in the Work buffer.

For example, if you delete a word from the middle of a sentence by using the dw command and then move the cursor to a SPACE between two words and give a p command, vim places the word you just deleted at the new location. If you delete a line using the dd command and then move the cursor to the line below the line where you want the deleted line to appear and give a P command, vim places the line at the new location.

Optional: Named Buffers

You can use a Named buffer with any of the Delete, Yank, or Put commands. Each of the 26 Named buffers is named by a letter of the alphabet. Each Named buffer can store a different block of text so that you can recall each block as needed. Unlike the General-Purpose buffer, vim does not change the contents of a Named buffer unless you use a command that specifically overwrites that buffer. The vim editor maintains the contents of the Named buffers throughout an editing session.

The vim editor stores text in a Named buffer if you precede a Delete or Yank command with a double quotation mark (") and a buffer name (for example, "kyy yanks a copy of the current line into buffer k). You can use a Named buffer in two ways. First, if you give the name of the buffer as a lowercase letter, vim overwrites the contents of the buffer when it deletes or yanks text into the buffer. Second, if you use an uppercase letter for the buffer name, vim appends the newly deleted or yanked text to the end of the buffer. This feature enables you to collect blocks of text from various sections of a file and deposit them at one place in the file with a single command. Named buffers are also useful when you are moving a section of a file and do not want to use a Put command immediately after the corresponding Delete command, and when you want to insert a paragraph, sentence, or phrase repeatedly in a document.

If you have one sentence that you use throughout a document, you can yank that sentence into a Named buffer and put it wherever you need it by using the following procedure: After entering the first occurrence of the sentence and pressing ESCAPE to return to Command mode, leave the cursor on the line containing the sentence. (The sentence must appear on a line or lines by itself for this procedure to work.) Then yank the sentence into Named buffer a by giving the "ayy command (or "a2yy if the sentence takes up two lines). Now anytime you need the sentence, you can return to Command mode and give the command "ap to put a copy of the sentence below the line the cursor is on.

This technique provides a quick and easy way to insert text that you use frequently in a document. For example, if you were editing a legal document, you might store the phrase The Plaintiff alleges that the Defendant in a Named buffer to save yourself the trouble of typing it every time you want to use it. Similarly, if you were creating a letter that frequently used a long company name, such as National Standards Institute, you might put it into a Named buffer.

NUMBERED BUFFERS

In addition to the 26 Named buffers and 1 General-Purpose buffer, 9 Numbered buffers are available. They are, in one sense, readonly buffers. The vim editor fills them with the nine most recently deleted chunks of text that are at least one line long. The most recently deleted pattern is held in "1, the next most recent in "2, and so on. If you delete a block of text and then give other vim commands so that you cannot reclaim the deleted text with Undo, use "1p to paste the most recently deleted chunk of text below the location of the cursor. If you have deleted several blocks of text and want to reclaim a specific one, proceed as follows: Paste the contents of the first buffer with "1p. If the first buffer does not have the text you are looking for, undo the paste with u and then give the period (.) command to repeat the previous command. The Numbered buffers work in a unique way with the period command: Instead of pasting the contents of buffer "1, the period command pastes the contents of the next buffer ("2). Another u and period replace the contents of buffer "2 with that of buffer "3, and so on through the nine buffers.


Reading and Writing Files

Exit (ZZ)

The vim editor reads a disk file into the Work buffer when you specify a filename on the command line you use to call vim. The ZZ command that terminates the editing session writes the contents of the Work buffer back to the disk file. This section discusses other ways of reading text into the Work buffer and writing it to a file.

Reading Files

Read (:r)

The Read command reads a file into the Work buffer. The new file does not overwrite any text in the Work buffer but rather is positioned following the single address you specify (or the current line if you do not specify an address). You can use an address of 0 to read the file into the beginning of the Work buffer. The Read command has the following syntax:

:[address]r [filename]

As with other commands that begin with a colon, when you enter the colon it appears on the status line. The filename is the pathname of the file that you want to read and must be terminated by RETURN. If you omit the filename, vim reads the file you are editing from the disk.

Writing Files

Write (:w)

The Write command writes part or all of the Work buffer to a file. You can use an address to write out part of the Work buffer and a filename to specify a file to receive the text. If you do not use an address or filename, vim writes the entire contents of the Work buffer to the file you are editing, updating the file on the disk.

During a long editing session, it is a good idea to use the Write command occasionally. If a problem later develops, a recent copy of the Work buffer is then safe on the disk. If you use a :q! command to exit from vim, the disk file reflects the version of the Work buffer at the time you last used the Write command. The Write command has two possible formats:

:[address]w[!] [filename]
:[address]w>> filename

The second format appends text to an existing file. The address specifies the portion of the Work buffer vim will write to the file. The address follows the form of the address that the Substitute command uses (page 172). If you do not specify an address, vim writes the entire contents of the Work buffer. The optional filename is the pathname of the file you are writing to. If you do not specify a filename, vim writes to the file you are editing.

w!

Because the Write command can quickly destroy a large amount of work, vim demands that you enter an exclamation point (!) following the w as a safeguard against accidentally overwriting a file. The only times you do not need an exclamation point are when you are writing out the entire contents of the Work buffer to the file being edited (using no address and no filename) and when you are writing part or all of the Work buffer to a new file. When you are writing part of the file to the file being edited or when you are overwriting another file, you must use an exclamation point.

Identifying the Current File

The File command (:f) provides the same information as the Status command (CONTROL-G, page 175). The filename the File command displays is the one the Write command uses if you give a :w command without a filename.


Setting Parameters

You can tailor the vim editor to your unique needs and habits by setting vim parameters. Parameters perform such functions as displaying line numbers, automatically inserting RETURNs for you, and establishing incremental and nonstandard searches.

You can set parameters in several ways. For example, you can set them to establish the environment for the current editing session while you are using vim. Alternatively, you can set the parameters in your ~/.bash_profile (bash) or ~/.login (tcsh) shell startup file or in the vim startup file, ~/.vimrc. When you set the parameters in any of these files, each time you use vim, the environment has been established and you can begin editing immediately.

Setting Parameters from Within vim

To set a parameter while you are using vim, enter a colon (:), the word set, a SPACE, and the parameter (refer to "Parameters" on page 181). The command appears on the status line as you type it and takes effect when you press RETURN. The following command establishes incremental searches for the current editing session:

:set incsearch

Setting Parameters in a Startup File

VIMINIT

If you are using bash, you can put a line with the following format in your ~/.bash_profile startup file (page 259):

export VIMINIT='set param1 param2 ...'

Replace param1 and param2 with parameters selected from Table 6-7. VIMINIT is a shell variable that vim reads. The following statement ignores the case of characters in searches, displays line numbers, uses the TC Shell to execute shell commands, and wraps text 15 characters from the right edge of the screen:

export VIMINIT='set autoindent number shell=/bin/tcsh wrapmargin=15'

Table 6-7. Parameters

Parameter

Effect

Allow special characters in searches
magic

Refer to "Special Characters in Search Strings" on page 170. By default the following characters have special meanings when used in a search string:

. [ ] *

When you set the nomagic parameter, these characters no longer have special meanings. The magic parameter restores their special meanings.

The ^ and $ characters always have special meanings within search strings, regardless of how you set this parameter.

Automatic indention
autoindent, ai

The automatic indention feature works with the shiftwidth parameter to provide a regular set of indentions for programs or tabular material. This feature is off by default. You can turn it on by setting autoindent and turn it off by setting noautoindent.

When automatic indention is on and vim is in Input mode, CONTROL-T moves the cursor from the left margin (or an indention) to the next indention position, RETURN moves the cursor to the left side of the next line under the first character of the previous line, and CONTROL-D backs up over indention positions. The CONTROL-T and CONTROL-D keys work only before text is placed on a line.

Automatic write
autowrite, aw

By default vim asks you before writing out the Work buffer when you have not explicitly told it to do so (as when you give a :n command to edit the next file). The autowrite option causes vim to write the Work buffer automatically when you use commands, such as :n, to edit to another file. You can disable this parameter by setting the noautowrite or noaw option.

Flash
flash, fl

The vim editor normally causes the terminal to beep when you give an invalid command or press ESCAPE when it is in Command mode. Setting the parameter flash causes the terminal to flash instead of beep. Set noflash to cause it to beep. Not all terminals and emulators support this parameter.

Ignore case in searches
ignorecase, ic

The vim editor normally performs case-sensitive searches, differentiating between uppercase and lowercase letters. It performs case-insensitive searches when you set the ignorecase parameter. Set noignorecase to restore case-sensitive searches.

Incremental search
incsearch, is

Refer to "Normal Versus Incremental Searches" on page 169. By default vim does not perform incremental searches. To cause vim to perform incremental searches, set the parameter incsearch. To cause vim not to perform incremental searches, set the parameter noincsearch.

Invisible characters
list

To cause vim to display each TAB as ^l and to mark the end of each line with a $, set the list parameter. To display TABs as whitespace and not mark ends of lines, set nolist.

Status line
laststatus=n, Is=n

Displays a status line that shows the name of the file you are editing, a [+] if the file has been changed since it was last written out, and the position of the cursor. Set the parameter laststatus=n, where n is 0 (zero) to turn off the status line, 1 to display the status line when at least two windows are displayed, or 2 to always display the status line.

Line numbers
number, nu

The vim editor does not normally display the line number associated with each line. To display line numbers, set the parameter number. To cause line numbers not to be displayed, set the parameter nonumber.

Line numbers are not part of the file, are not stored with the file, and are not displayed when the file is printed. They appear on the screen only while you are using vim.

Line wrap
wrap

The line wrap controls how vim displays lines that are too long to fit on the screen. To cause vim to wrap long lines and continue them on the next line, set wrap (set by default). If you set nowrap, vim truncates long lines at the right edge of the screen.

Line wrap margin
wrapmargin=nn, wm=nn

The line wrap margin causes vim to break the text that you are inserting at approximately the specified number of characters from the right margin. The vim editor breaks the text by inserting a NEWLINE character at the closest blank-delimited word boundary. Setting the line wrap margin is handy if you want all your text lines to be about the same length. It relieves you of having to remember to press RETURN to end each line of input.

Set the parameter wrapmargin=nn, where nn is the number of characters from the right side of the screen where you want vim to break the text. This number is not the column width of the text but the distance from the end of the text to the right edge of the screen. Setting the wrap margin to 0 (zero) turns this feature off. By default the line wrap margin is off (set to 0).

Report
report=nn

Causes vim to display a report on the status line whenever you make a change that affects at least nn lines. For example, if report is set to 7 and you delete seven lines, vim displays the message 7 lines deleted. When you delete six or fewer lines, vim does not display a message. The default for report is 5.

Scroll
scroll=nn, scr=nn

Controls the number of lines that CONTROL-D and CONTROL-U (page 162) scroll text on the screen. By default scroll is set to half the window height.

There are two ways to change the value of scroll. First you can enter a number before giving a CONTROL-D or CONTROL-U command; vim sets scroll to that number. Alternatively, you can set scroll explicitly with scroll=nn, where nn is the number of lines you want to scroll with each CONTROL-D or CONTROL-U command.

Shell
shell=path, sh=path

While you are using vim, you can cause it to spawn a new shell. You can either create an interactive shell (if you want to run several commands) or run a single command. The shell parameter determines which shell vim invokes. By default vim sets the shell parameter to your login shell. To change it, set the parameter shell=path, where path is the absolute pathname of the shell you want to use.

Shift width
shiftwidth=nn, sw=nn

Controls the functioning of CONTROL-T and CONTROL-D in Input mode when automatic indention is on (see "Automatic indention" in this table). Set the parameter shiftwidth=nn, where nn is the spacing of the indention positions (8 by default). Setting the shift width is similar to setting the TAB stops on a typewriter; with shiftwidth, however, the distance between TAB stops remains constant.

Show match
showmatch, sm

Useful for programmers working in languages that use braces ({}) or parentheses as expression delimiters (Lisp, C, Tcl, and so on). When showmatch is set and you are entering code (in Input mode) and type a closing brace or parenthesis, the cursor jumps briefly to the matching opening brace or parenthesis (that is, the preceding corresponding element at the same nesting level). After it highlights the matching element, the cursor resumes its previous position. When you type a right brace or parenthesis that does not have a match, vim beeps. Use noshowmatch to turn off automatic matching.

Show mode
showmode, smd

Set the parameter showmode to display the mode in the lower-right corner of the screen when vim is in Input mode (default). Set noshowmode to cause vim not to display the mode.

vi compatibility
compatible, cp

Refer to "The compatible Parameter" on page 153. By default except when you have a .vimrc startup file (page 180), vim attempts to be compatible with vi. To cause vim to be compatible with vi, set the parameter compatible. To cause vim not to be compatible with vi, set the parameter nocompatible.

Wrap scan
wrapscan, ws

By default when a search for the next occurrence of a search string reaches the end of the Work buffer, vim continues the search at the beginning of the Work buffer. The reverse is true of a search for the previous occurrence of a search string. The nowrapscan parameter stops the search at either end of the Work buffer. Set the wrapscan parameter if you want searches to wrap around the ends of the Work buffer.

If you use the parameter abbreviations, it looks like this:

export VIMINIT='set ai nu sh=/bin/tcsh wm=15'

If you are using tcsh, put the following line in your ~/.tcshrc startup file (page 340).

setenv VIMINIT 'set param1 param2 ...'

Again, replace param1 and param2 with parameters from Table 6-7. The values between the single quotation marks are the same as shown in the preceding examples.

The .vimrc Startup File

Instead of setting vim parameters in your shell startup file, you can create a ~/.vimrc file in your home directory and set them there. Creating a .vimrc file causes vim to start with the compatible parameter unset (page 153). Lines in a .vimrc file use the following format:

set param1 param2 ...

Following are examples of .vimrc files that perform the same function as VIMINIT described previously:

$ cat ~/.vimrc
set ignorecase
set number
set shell=/bin/tcsh
set wrapmargin=15

$ cat ~/.vimrc
set ic nu sh=/bin/tcsh wm=15

Parameters set by the VIMINIT variable take precedence over those set in the .vimrc file.

Parameters

Table 6-7 lists some of the most useful vim parameters. The vim editor displays a complete list of parameters and indicates how they are currently set when you give the command :set all followed by a RETURN. The command :set RETURN displays a list of options that are set to values other than their default values. Two classes of parameters exist: those that contain an equal sign (and can take on a value) and those that are optionally prefixed with no (switches that are on or off). You can change the sense of a switch parameter by giving the command :set [no]param. For example, give the command :set number (or :set nonumber) to turn on (or off) line numbering. To change the value of a parameter that takes on a value (and uses an equal sign), give a command such as :set shiftwidth=15.

Most parameters have abbreviations such as nu for number, nonu for no number, and sw for shiftwidth. The abbreviations are listed in the left column of Table 6-7, following the name of the parameter.


Advanced Editing Techniques

This section presents several commands that you may find useful once you have become comfortable using vim.

Optional: Using Markers

While you are using vim, you can set and use markers to make addressing more convenient. Set a marker by giving the command mc, where c is any character. (Letters are preferred because some characters, such as a single quotation mark, have special meanings when used as markers.) The vim editor does not preserve markers when you exit from vim.

Once you have set a marker, you can use it in a manner similar to a line number. You can move the cursor to the beginning of a line that contains a marker by preceding the marker name with a single quotation mark. For example, to set marker t, position the cursor on the line you want to mark and give the command mt. During this editing session, unless you reset marker t or delete the line it marks, you can return to the beginning of the line you marked with the command 't.

You can delete all text from the current line through the line containing marker r with the following command:

d'r

You can use a back tick (`, also called a grave accent or reverse single quotation mark) to go to the exact position of the mark on the line. After setting marker t, you can move the cursor to the location of this marker (not the beginning of the line that holds the marker) with the command `t. The following command deletes all the text from the current line up to the character where the mark r was placed; the rest of the line containing the marker remains intact:

d`r

You can use markers in addresses of commands instead of line numbers. The following command replaces all occurrences of The with THE on all lines from marker m to the current line (marker m must precede the current line):

:'m,.s/The/THE/g

EDITING OTHER FILES

The following command causes vim to edit the file you specify with filename:

:e[!] [filename]

If you want to save the contents of the Work buffer, you must write it out (using :w) before you give this command. If you do not want to save the contents of the Work buffer, vim insists that you use an exclamation point to acknowledge that you will lose the work you did since the last time you wrote out the Work buffer. If you do not supply a filename, vim edits the same file you are currently working on.

:e!

The command :e! starts an editing session over again. This command returns the Work buffer to the state it was in the last time you wrote it out or, if you have not written it out, the state it was in when you started editing the file. It is useful when you make mistakes while editing a file and decide that it would be easier to start over than to fix the mistakes.

Because this command does not destroy the contents of the General-Purpose or Named buffers, you can store text from one file in a buffer, use a :e command to edit a second file, and put text from the buffer in the second file.

:e#

The command :e# closes the current file and opens the last file you were editing, placing the cursor on the line that it was on when you last closed the file. If you do not save the file you are working on before you give this command, vim prompts you to do so. Setting the autowrite parameter (page 181) will not stop vim from prompting you.

:n :rew

The :e# command can help you copy blocks of text from one file to another. Call vim with the names of several files as arguments. You can use :n to edit the next file, :e# to edit the file you just edited, and :rew to rewind the sequence of files so that you are editing the first file again. As you move between files, you can copy text from one file into a buffer and paste that text into another file. You can use :n! to force vim to close a file without writing out changes before it opens the next file.

MACROS AND SHORTCUTS

:map

The vim editor allows you to create macros and shortcuts. The :map command defines a key or sequence of keys that perform some action in Command mode. The following command maps CONTROL-X to the commands that will find the next left bracket on the line the cursor is on (f[), delete all characters from that bracket to the next right bracket (df]) on the same line, delete the next character (x), move the cursor down two lines (2j), and finally move the cursor to the beginning of the line (0):

:map ^X f[df]x2j0

You can use ESCAPE and CONTROL sequences but try to avoid remapping characters or sequences that are vim commands. Type :map by itself to see a list of the current mappings. You may need to use CONTROL-V (page 164) to quote some of the characters you want to enter into the :map string.

:abbrev

The :abbrev command is similar to :map but creates abbreviations you can use while in Input mode. When you are in Input mode and type a string you have defined with :abbrev, followed by a SPACE, vim replaces the string and the SPACE with the characters you specified when you defined the string. For ease of use, do not use common sequences of characters when creating abbreviations. The following command defines ZZ as an abbreviation for Mark G. Sobell:

:abbrev ZZ Mark G. Sobell

Even though ZZ is a vim command, it is used only in Command mode. It has no special meaning in Input mode, where you use abbreviations.

Executing Shell Commands from Within vim

You can execute shell commands in several ways while you are using vim. You can spawn a new interactive shell by giving the following command and pressing RETURN:

:sh

The vim shell parameter (page 183) determines which shell is spawned (usually bash or tcsh). By default shell is the same as your login shell.

After you have finished your work in the shell, you can return to vim by exiting from the shell (press CONTROL-D or give an exit command).

Tip: If :sh does not work correctly

The :sh command may behave strangely depending on how your shell has been configured. You may get warnings with the :sh command or it may even hang. Experiment with the :sh command to be sure it works with your configuration. If it does not, then you might want to set the vim shell parameter to another shell before using :sh. For example, the following command causes vim to use tcsh with the :sh command:

:set shell=/bin/tcsh

You may need to change the SHELL environment variable after starting :sh to show the correct shell.

Caution: Edit only one copy of a file

When you create a new shell by using :sh, remember that you are still using vim. A common mistake is to try to edit the same file from the new shell, forgetting that vim is already editing the file from a different shell. Because you can lose information by editing the same file from two instances of an editor, vim warns you when you make this mistake. Refer to "File Locks" on page 157 to see an example of the message that vim displays.

You can execute a shell command line from vim by giving the following command, replacing command with the command line you want to execute and terminating the command with a RETURN:

:!command

The vim editor spawns a new shell that executes the command. When the command runs to completion, the newly spawned shell returns control to the editor.

You can execute a command from vim and have it replace the current line with the output from the command. If you do not want to replace any text, put the cursor on a blank line before giving the following command:

!!command

Nothing happens when you enter the first exclamation point. When you enter the second one, vim moves the cursor to the status line and allows you to enter the command you want to execute. Because this command puts vim in Last Line mode, you must end the command with a RETURN (as you would end most shell commands).

Finally you can execute a command from vim with standard input to the command coming from all or part of the file you are editing and standard output from the command replacing the input in the file you are editing. You can use this type of command to sort a list in place within a file.

To specify the block of text that will become standard input for the command, move the cursor to one end of the block of text. Then enter an exclamation point followed by a command that would normally move the cursor to the other end of the block of text. For example, if the cursor is at the beginning of the file and you want to specify the whole file, give the command !G. If you want to specify the part of the file between the cursor and marker b, give the command !'b. After you give the cursor-movement command, vim displays an exclamation point on the status line and waits for you to enter a shell command.

To sort a list of names in a file, move the cursor to the beginning of the list and set marker q with an mq command. Then move the cursor to the end of the list and give the following command:

!'sort

Press RETURN and wait. After a few seconds, the sorted list should replace the original list on the screen. If the command did not do what you expected, you can usually undo the change with a u command. Refer to page 837 for more information on sort.

Caution: ! can destroy a file

If you enter the wrong command or mistype a command, you can destroy a file (for example, if the command hangs or stops vim from working). For this reason it is a good idea to save your file before using this command. The Undo command (page 165) can be a lifesaver. A :e! command (page 185) will get rid of your changes, returning the buffer to the state it was in last time you saved it.

As with the :sh command, the default shell may not work properly with the ! command. You may want to test the shell with a simple file before executing this command with your real work. If the usual shell does not work properly, change the shell parameter.


Units of Measure

Many vim commands operate on a block of textranging from one character to many paragraphs. You specify the size of a block of text with a Unit of Measure. You can specify multiple Units of Measure by preceding a Unit of Measure with a Repeat Factor (page 191). This section defines the various Units of Measure.

Character

A character is one charactervisible or not, printable or notincluding SPACEs and TABs. Some examples of characters are

a q A . 5 R - > TAB SPACE

Word

A word, similar to an ordinary word in the English language, is a string of one or more characters bounded on both sides by any combination of one or more of the following elements: a punctuation mark, SPACE, TAB, numeral, or NEWLINE. In addition, vim considers each group of punctuation marks to be a word (Table 6-8).

Table 6-8. Words

Word count

Text

1

pear

2

pear!

2

pear!)

3

pear!) The

4

pear!) "The

11

This is a short, concise line (no frills).

Blank-Delimited Word

A blank-delimited word is the same as a word but includes adjacent punctuation. Blank-delimited words are separated by one or more of the following elements: a SPACE, TAB, or NEWLINE (Table 6-9).

Table 6-9. Blank-delimited words

Word count

Text

1

pear

1

pear!

1

pear!)

2

pear!) The

2

pear!) "The

8

This is a short, concise line (no frills).

Line

A line is a string of characters bounded by NEWLINEs that is not necessarily displayed as a single physical line on the screen. You can enter a very long single (logical) line that wraps around (continues on the next physical line) several times or disappears off the right edge of the display. It is a good idea to avoid long logical lines by terminating lines with a RETURN before they reach the right side of the screen. Terminating lines in this manner ensures that each physical line contains one logical line and avoids confusion when you edit and format text. Some commands do not appear to work properly on physical lines that are longer than the width of the screen. For example, with the cursor on a long logical line that wraps around several physical lines, pressing RETURN once appears to move the cursor down more than one line. You can use fmt (page 736) to break long logical lines into shorter ones.

Sentence

A sentence is an English sentence or the equivalent. A sentence starts at the end of the previous sentence and ends with a period, exclamation point, or question mark, followed by two SPACEs or a NEWLINE (Table 6-10).

Table 6-10. Sentences

Sentence count

Text

One: only one SPACE after the first period and a NEWLINE after the second period

That's it. This is one sentence.

Two: two SPACEs after the first period and a NEWLINE after the second period

That's it. This is two sentences.

Three: two SPACEs after the first two question marks and a NEWLINE after the exclamation point

What? Three sentences? One line!

One: NEWLINE after the period

This sentence takes
up a total of
three lines.

Paragraph

A paragraph is preceded and followed by one or more blank lines. A blank line is composed of two NEWLINE characters in a row (Table 6-11).

Table 6-11. Paragraphs

Paragraph count

Text

One: blank line before and after text

One paragraph

One: blank line before and after text

       This may appear to be
more than one paragraph.
       Just because there are
two indentions does not mean
it qualifies as two paragraphs.

Three: three blocks of text separated by blank lines

Even though in

English this is only
one sentence,

vim considers it to be
three paragraphs.

Window

Under vim, a screen or terminal emulator window can display one or more logical windows of information. A window displays all or part of a Work buffer. Figure 6-5 on page 151 shows a screen with two windows.

Repeat Factor

A number that precedes a Unit of Measure (page 188) is a Repeat Factor. Just as the 5 in 5 inches causes you to consider 5 inches as a single Unit of Measure, so a Repeat Factor causes vim to group more than one Unit of Measure and consider it as a single Unit of Measure. For example, the command w moves the cursor forward 1 word, the command 5w moves it forward 5 words, and the command 250w moves it forward 250 words. If you do not specify a Repeat Factor, vim assumes a Repeat Factor of 1. If the Repeat Factor would move the cursor past the end of the file, the cursor is left at the end of the file.


Chapter Summary

This summary of vim includes all the commands covered in this chapter, plus a few more. Table 6-12 lists some of the ways you can call vim from the command line.

Table 6-12. Calling vim

Command

Result

vim filename

Edits filename starting at line 1

vim +n filename

Edits filename starting at line n

vim + filename

Edits filename starting at the last line

vim +/pattern filename

Edits filename starting at the first line containing pattern

vim r filename

Recovers filename after a system crash

vim R filename

Edits filename readonly (same as opening the file with view)

You must be in Command mode to use commands that move the cursor by Units of Measure (Table 6-13). You can use these Units of Measure with Change, Delete, and Yank commands. Each of these commands can be preceded by a Repeat Factor.

Table 6-13. Moving the cursor by Units of Measure

Command

Moves the cursor

SPACE, l (lowercase "l"), or RIGHT ARROW

Space to the right

h or LEFT ARROW

Space to the left

w

Word to the right

W

Blank-delimited word to the right

b

Word to the left

B

Blank-delimited word to the left

$

End of line

e

End of word to the right

E

End of blank-delimited word to the right

0 (zero)

Beginning of line (cannot be used with a Repeat Factor)

RETURN

Beginning of next line

j or DOWN ARROW

Down one line

Beginning of previous line

k or UP ARROW

Up one line

)

End of sentence

(

Beginning of sentence

}

End of paragraph

{

Beginning of paragraph

%

Move to matching brace of same type at same nesting level

Table 6-14 shows the commands that enable you to view different parts of the Work buffer.

Table 6-14. Viewing the Work buffer

Command

Moves the cursor

CONTROL-D

Forward one-half window

CONTROL-U

Backward one-half window

CONTROL-F or PAGE DOWN

Forward one window

CONTROL-B or PAGE UP

Backward one window

nG

To line n (without n, to the last line)

H

To top of window

M

To middle of window

L

To bottom of window

The commands in Table 6-15 enable you to add text to the buffer. All these commands, except r, leave vim in Input mode. You must press ESCAPE to return to Command mode.

Table 6-15. Adding text

Command

Adds text

i

Before cursor

I

Before first nonblank character on line

a

After cursor

A

At end of line

o

Open a line below current line

O

Open a line above current line

r

Replace current character (no ESCAPE needed)

R

Replace characters, starting with current character (overwrite until ESCAPE)

Table 6-16 lists commands that delete and change text. In this table M is a Unit of Measure that you can precede with a Repeat Factor, n is an optional Repeat Factor, and c is any character.

Table 6-16. Deleting and changing text

Command

Result

nx

Deletes the number of characters specified by n, starting with the current character

nX

Deletes n characters before the current character, starting with the character preceding the current character

dM

Deletes text specified by M

ndd

Deletes n lines

dtc

Deletes to the next character c on the current line

D

Deletes to end of the line

n~

Change case of the next n characters

The following commands leave vim in Input mode. You must press ESCAPE to return to Command mode.

ns

Substitutes n characters

S

Substitutes for the entire line

cM

Changes text specified by M

ncc

Changes n lines

ctc

Changes to the next character c on the current line

C

Changes to end of line

Table 6-17 lists search commands. Here rexp is a regular expression that can be a simple string of characters.

Table 6-17. Searching

Command

Result

/rexp RETURN

Searches forward for rexp

?rexp RETURN

Searches backward for rexp

n

Repeats original search exactly

N

Repeats original search, in the opposite direction

/RETURN

Repeats original search forward

?RETURN

Repeats original search backward

fc

Positions the cursor on the next character c on the current line

Fc

Positions the cursor on the previous character c on the current line

tc

Positions the cursor on the character before (to the left of) the next character c on the current line

Tc

Positions the cursor on the character after (to the right of) the previous character c on the current line

;

Repeats the last f, F, t, or T command

The format of a Substitute command is

:[address]s/search-string/replacement-string[/g]

where address is one line number or two line numbers separated by a comma. A . (period) represents the current line, $ represents the last line, and % represents the entire file. You can use a marker or a search string in place of a line number. The search-string is a regular expression that can be a simple string of characters. The replacement-string is the replacement string. A g indicates a global replacement (more than one replacement per line).

Table 6-18 lists miscellaneous vim commands.

Table 6-18. Miscellaneous commands

Command

Result

J

Joins the current line and the following line

.

Repeats the most recent command that made a change

:w filename

Writes contents of Work buffer to filename (or to current file if there is no filename)

:q

Quits vim

ZZ

Writes contents of Work buffer to the current file and quits vim

:f or CONTROL-G

Displays the filename, status, current line number, number of lines in the Work buffer, and percentage of the Work buffer preceding the current line

CONTROL-V

Inserts the next character literally even if it is a vim command (use in Input mode)

Table 6-19 lists commands that yank and put text. In this table M is a Unit of Measure that you can precede with a Repeat Factor and n is a Repeat Factor. You can precede any of these commands with the name of a buffer using the form "x, where x is the name of the buffer (az).

Table 6-19. Yanking and putting text

Command

Result

yM

Yanks text specified by M

nyy

Yanks n lines

Y

Yanks to end of line

P

Puts text before or above

p

Puts text after or below

Table 6-20 lists advanced vim commands.

Table 6-20. Advanced commands

Command

Result

mx

Sets marker x, where x is a letter from a to z.

'' (two single quotation marks)

Moves cursor back to its previous location.

'x

Moves cursor to line with marker x.

` x

Moves cursor to character with marker x.

:e filename

Edits filename, requiring you to write out changes to the current file (with :w or autowrite) before editing the new file. Use :e! filename to discard changes to the current file. Use :e! without a filename to discard changes to the current file and start editing the saved version of the current file.

:n

Edits the next file when vim is started with multiple filename arguments. Requires you to write out changes to the current file (with :w or autowrite) before editing the next file. Use :n! to discard changes to the current file and edit the next file.

:rew

Rewinds the filename list when vim is started with multiple filename arguments and starts editing with the first file. Requires you to write out changes to the current file (with :w or autowrite) before editing the first file. Use :rew! to discard changes to the current file and edit the first file.

:sh

Starts a shell. Exit from the shell to return to vim.

:!command

Starts a shell and executes command.

!!command

Starts a shell, executes command, and places output in the Work buffer, replacing the current line.


Exercises

1.

How can you cause vim to enter Input mode? How can you make vim revert to Command mode?

2.

What is the Work buffer? Name two ways of writing the contents of the Work buffer to the disk.

3.

Suppose that you are editing a file that contains the following paragraph and the cursor is on the second tilde (~):

The vim editor has a command, tilde (~),
that changes lowercase letters to
uppercase, and vice versa.
The ~ command works with a Unit of Measure or
a Repeat Factor, so you can change
the case of more than one character at a time.

How can you

  1. Move the cursor to the end of the paragraph?

  2. Move the cursor to the beginning of the word Unit?

  3. Change the word character to letter?

4.

While working in vim, with the cursor positioned on the first letter of a word, you give the command x followed by p. Explain what happens.

5.

What are the differences between the following commands?

  1. i and I

  2. a and A

  3. o and O

  4. r and R

  5. u and U

6.

Which command would you use to search backward through the Work buffer for lines that start with the word it?

7.

Which command substitutes all occurrences of the phrase this week with the phrase next week?

8.

Consider the following scenario: You start vim to edit an existing file. You make many changes to the file and then realize that you deleted a critical section of the file early in your editing session. You want to get that section back but do not want to lose all the other changes you made. What would you do?

9.

How can you move the line that the cursor is on to the beginning of the file?

10.

Use vim to create the letter_e file of e's used on page 54. Use as few vim commands as possible. Which vim commands did you use?


Advanced Exercises

11.

Which commands can you use to take a paragraph from one file and insert it in a second file?

12.

Create a file that contains the following list, and then execute commands from within vim to sort the list and display it in two columns. (Hint: Refer to page 817 for more information on pr.)

Command mode
Input mode
Last Line mode
Work buffer
General-Purpose buffer
Named buffer
Regular Expression
Search String
Replacement String
Startup File
Repeat Factor

13.

How do the Named buffers differ from the General-Purpose buffer?

14.

Assume that your version of vim does not support multiple Undo commands. If you delete a line of text, then delete a second line, and then a third line, which commands would you use to recover the first two lines that you deleted?

15.

Which command would you use to swap the words hither and yon on any line with any number of words between them? (You need not worry about special punctuation, just uppercase and lowercase letters and spaces.)


Chapter 7. The emacs Editor

Previous Page
Next Page

7. The emacs Editor

IN THIS CHAPTER

History | 200
emacs Versus vim | 201
Tutorial: Getting Started with emacs | 201
Basic Editing Commands | 207
Online Help | 213
Advanced Editing | 215
Language-Sensitive Editing | 228
Customizing emacs | 238

In 1956 the Lisp (List processing) language was developed at MIT by John McCarthy. In its original conception, Lisp had only a few scalar (atomic) data types and only one data structure (page 929): a list. Lists could contain atomic data or perhaps other lists. Lisp supported recursion and nonnumeric data (exciting concepts in those FORTRAN and COBOL days) and, in the Cambridge culture at least, was once the favored implementation language. Richard Stallman and Guy Steele were part of this MIT Lisp culture. In 1975 they collaborated on emacs, which Stallman maintained by himself for a long time. This chapter discusses the emacs editor as implemented by the Free Software Foundation (GNU). The emacs home page is www.gnu.org/software/emacs.


History

The emacs editor was prototyped as a series of extension commands or macros for the late 1960s text editor TECO (Text Editor and COrrector). Its acronymic name, Editor MACroS, reflects this origin, although there have been many humorous reinterpretations, including ESCAPE META ALT CONTROL SHIFT, Emacs Makes All Computing Simple, and the unkind translation Eight Megabytes And Constantly Swapping.

Evolution

Over time emacs has grown and evolved through more than 20 major revisions to the mainstream GNU version. The emacs editor, which is coded in C, contains a complete Lisp interpreter and fully supports the X Window System and mouse interaction. The original TECO macros are long gone, but emacs is still very much a work in progress. There are plans to support variable-width fonts, wide character sets, and the world's major languages as well as to move emacs in the direction of a WYSIWYG (what you see is what you get) word processor and make it easier for beginners to use.

The emacs editor has always been considerably more than a text editor. Not having been developed originally in a UNIX environment, it does not adhere to the UNIX philosophy. Whereas a UNIX utility is typically designed to do one thing and to be used in conjunction with other utilities, emacs is designed to "do it all." Taking advantage of the underlying programming language (Lisp), emacs users tend to customize and extend the editor rather than to use existing utilities or create new general-purpose tools. Instead they share their ~/.emacs (customization) files.

Well before the emergence of the X Window System, Stallman put a great deal of thought and effort into designing a window-oriented work environment, and he used emacs as his research vehicle. Over time he built facilities within emacs for reading and composing email messages, reading and posting netnews, giving shell commands, compiling programs and analyzing error messages, running and debugging these programs, and playing games. Eventually it became possible to enter the emacs environment and not come out all day, switching from window to window and from file to file. If you had only an ordinary serial, character-based terminal, emacs gave you tremendous leverage.

In an X Window System environment, or on the Mac OS X desktop, emacs does not need to control the whole display. Instead, it usually operates only one or two windows. The original work environment is still available and is covered in this chapter.

As a language-sensitive editor, emacs has special features that you can turn on to help edit text, nroff, TeX, Lisp, C, Fortran, and so on. These feature sets are called modes, but they are not related in any way to the Command mode and Input mode found in vi, vim, and other editors. Because you never need to switch emacs between Input and Command modes, emacs is a modeless editor.

emacs Versus vim

Like vim, emacs is a display editor: It displays on the screen the text you are editing and changes the display as you type each command or insert new text. Unlike vim, emacs does not require you to keep track of whether you are in Command mode or Insert mode: Commands always use CONTROL or other special keys. The emacs editor inserts ordinary characters into the text you are editing (as opposed to using ordinary characters as commands), another trait of modeless editing. For many people this approach is convenient and natural.

Like vim, emacs has a rich, extensive command set for moving about in the buffer and altering text. This command set is not "cast in concrete"you can change or customize commands at any time. Literally any key can be coupled (bound) to any command so as to match a particular keyboard better or just to fulfill a personal whim. Usually key bindings are set in the .emacs startup file, but they can also be changed interactively during a session. All the key bindings described in this chapter are standard on current GNU emacs versions, which also support many visual, mouse-oriented capabilities that are not covered here.

Caution: Too many key bindings

If you change too many key bindings, you may produce a command set that you will not remember or that will make it impossible for you to get back to the standard bindings again in the same session.

Finally, and very unlike vim, emacs allows you to use Lisp to write new commands or override old ones. Stallman calls this feature online extensibility, but it would take a gutsy Lisp guru to write and debug a new command while editing live text. It is much more common to add a few extra debugged commands to the .emacs file, where they are loaded when emacs starts up. Experienced emacs users often write modes, or environments, that are conditionally loaded by emacs for specific tasks.

Tip: The screen and emacs windows

In this chapter, the term screen denotes a character-based terminal screen or a terminal emulator window in a graphical environment. The term window refers to an emacs window within a screen.


Tutorial: Getting Started with emacs

The emacs editor has many, many features, and there are many ways to use it. Its complete manual includes more than 35 chapters. Nevertheless, you can do a considerable amount of meaningful work with a relatively small subset of the commands. This section describes a simple editing session, explaining how to start and exit from emacs and how to move the cursor and delete text. Some issues are postponed or simplified in the interest of clarity.

Starting emacs

To edit a file named sample using emacs as a text-based editor, enter the following command. The nw option, which must be the first option on the emacs command line, tells emacs not to use its X (GUI) interface.

$ emacs -nw -q sample

This command starts emacs, reads the file named sample into a buffer, and displays its contents on the screen or window. If no file has this name, emacs displays a blank screen with New File at the bottom (Figure 7-1). If the file exists, emacs displays another message (Figure 7-2, page 204). The q option tells emacs not to read the ~/.emacs startup file from your home directory. Not reading the startup file guarantees that you get standard, uncustomized behavior and is sometimes useful for beginners or for other users who want to bypass a .emacs file.

Figure 7-1. The emacs welcome screen

Figure 7-2. Sample buffer

[View full size image]

The screen starts with a single window. At the bottom of this window is a reverse-video titlebar called the Mode Line. At a minimum, the Mode Line shows which buffer the window is viewing, whether the buffer has been changed, what major and minor modes are in effect, and how far down the buffer the window is positioned. When you have more than one window, one Mode Line appears in each window. At the bottom of the screen, emacs leaves a single line open. This Echo Area, or Minibuffer, line is used for short messages and special one-line commands.

A cursor is in the window or Minibuffer. All input and nearly all editing takes place at the cursor. As you type ordinary characters, emacs inserts them at the cursor position. If characters are under the cursor or to its right, they get pushed to the right as you type, so no characters are lost.

Stopping emacs

The command to exit from emacs is the two-key sequence CONTROL-X CONTROL-C. You can give this command at almost any time (in some modes you may have to press CONTROL-G first). It stops emacs gracefully, asking you to confirm changes if you made any during the editing session.

If you want to cancel a half-typed command or stop a running command before it is done, you can quit by pressing CONTROL-G. The emacs editor displays Quit in the Echo Area and waits for your next command.

Inserting Text

Typing an ordinary (printing) character pushes the cursor and any characters to the right of the cursor one position to the right and inserts the new character in the position just opened. Backspacing pulls the cursor and any characters to the right of the cursor one position to the left, erasing the character that was there before.

Deleting Characters

Depending on the keyboard you are using and the emacs startup file, different keys may delete characters in different ways. CONTROL-D typically deletes the character under the cursor, as do DELETE and DEL. BACKSPACE typically deletes the character to the left of the cursor. Try each of these keys and see what it does.

Tip: More about deleting characters

If the instructions described in this section do not work, read the emacs info section on deletion. Give this command from a shell prompt:

$ info emacs

From info give the command m deletion to display a document that describes in detail how to delete small amounts of text. Use the SPACE bar to scroll through the document. Type q to exit from info.

Start emacs and type a few lines of text. If you make a mistake, use the deletion characters discussed previously. The RETURN key inserts an invisible end-of-line character in the buffer and returns the cursor to the left margin, one line down. It is possible to back up past the start of a line and up to the end of the previous line. Figure 7-2 shows a sample buffer.

Moving the Cursor

You can position the cursor over any character in the emacs window and move the window so it displays any portion of the buffer. You can move the cursor forward or backward through the text (Figure 6-8, page 160) by various textual unitsfor example, characters, words, sentences, lines, and paragraphs. Any of the cursor-movement commands can be preceded by a repetition count (CONTROL-U followed by a numeric argument), which causes the cursor to move that number of textual units through the text. Refer to page 209 for a discussion of numeric arguments.

Tip: Use the ARROW keys

Sometimes the easiest way to move the cursor is by using the LEFT, RIGHT, UP, and DOWN ARROW keys.

Moving the Cursor by Characters

CONTROL-F

Pressing the RIGHT ARROW key or CONTROL-F moves the cursor forward one character. If the cursor is at the end of a line, these commands wrap it to the beginning of the next line. The command CONTROL-U 7 CONTROL-F moves the cursor seven characters forward (to the right).

CONTROL-B

Pressing the LEFT ARROW key or CONTROL-B moves the cursor backward one character. The command CONTROL-U 7 CONTROL-B moves the cursor seven characters backward (to the left). CONTROL-B works in a manner similar to CONTROL-F (Figure 7-3).

Figure 7-3. Moving the cursor by characters

Moving the Cursor by Words

META-f

Pressing META-f moves the cursor forward one word. To press META-f hold down the META or ALT key while you press f. For more information see the tip "Activating the META key" on page 31. If you do not have either of these keys, press ESCAPE, release it, and then press f. This command leaves the cursor on the first character that is not part of the word the cursor started on. The command CONTROL-U 4 META-f moves the cursor forward one space past the end of the fourth word. See page 208 for more about keys.

Tip: The META key

You must configure Terminal to use the OPTION key as the META key. For more information see the tip "Activating the META key" on page 31.

META-b

Pressing META-b moves the cursor backward one word so the cursor is on the first letter of the word it started on. If the cursor was on the first letter of a word, META-b moves the cursor to the first letter of the preceding word. It works in a manner similar to META-f (Figure 7-4).

Figure 7-4. Moving the cursor by words

Moving the Cursor by Lines

CONTROL-A CONTROL-E CONTROL-P CONTROL-N

Pressing CONTROL-A moves the cursor to the beginning of the line it is on; CONTROL-E moves it to the end. Pressing the UP ARROW key or CONTROL-P moves the cursor up one line to the position directly above where the cursor started; the DOWN ARROW key or CONTROL-N moves it down. As with the other cursor-movement keys, you can precede CONTROL-P and CONTROL-N with CONTROL-U and a numeric argument to move up or down multiple lines. You can use pairs of these commands to move the cursor up to the beginning of the previous line, down to the end of the following line, and so on (Figure 7-5).

Figure 7-5. Moving the cursor by lines

Moving the Cursor by Sentences, Paragraphs, and Window Position

META-a, META-e META-{, META-}

Pressing META-a moves the cursor to the beginning of the sentence the cursor is on; META-e moves the cursor to the end. META-{ moves the cursor to the beginning of the paragraph the cursor is on; META-} moves it to the end. (Sentences and paragraphs are defined starting on page 230.) You can precede any of these commands with a repetition count (CONTROL-U and a numeric argument) to move the cursor that many sentences or paragraphs.

META-r

Pressing META-r moves the cursor to the beginning of the middle line of the window. You can precede this command with CONTROL-U and a line number (here CONTROL-U does not indicate a repetition count but a screen line number). The command CONTROL-U 0 META-r moves the cursor to the beginning of the top line (line zero) in the window. You can replace zero with the line number of the line you want to move the cursor to or with a minus sign (), in which case the cursor moves to the beginning of the last line of the window (Figure 7-6).

Figure 7-6. Moving the cursor by sentences, paragraphs, and window position

[View full size image]

Editing at the Cursor Position

With the cursor in the window you can type new text, pushing the existing text to the right. Entering new text requires no special commands once the cursor is positioned. If you type so many characters that the text in a line goes past the right edge of the window, emacs puts a backslash (\) near the right edge of the window and wraps the text to the next line. The backslash appears on the screen but is not saved as part of the file and is never printed. Although you can create an arbitrarily long line, some UNIX tools have problems with text files containing such lines. You can split a line at any point by positioning the cursor and pressing RETURN.

Deleting text

Pressing DELETE removes characters to the left of the cursor. The cursor and the remainder of the text on this line both move to the left each time you press DELETE. To join a line with the line above it, position the cursor on the first character of the second line and press DELETE.

Press CONTROL-D to delete the character under the cursor. The cursor remains stationary, but the remainder of the text on the line moves left to replace the deleted character. See the tip "More about deleting characters" on page 203 if either of these keys does not work as described here.

Saving and Retrieving the Buffer

No matter what happens to a buffer during an emacs session, the associated file does not change until you save the buffer. If you leave emacs without saving the buffer (this is possible if you are insistent enough), the file is not changed and the session's work is discarded.

Backups

As mentioned previously, emacs prompts you about unsaved changes to the buffer contents. As it writes a buffer's edited contents back to the file, emacs may optionally first make a backup of the original file contents. You can choose to make no backups, one level (default), or an arbitrary number of levels. The one-level backup filenames are formed by appending a ~ character to the original filename. The multilevel backups append .~n~ to the filename, where n is the sequential backup number, starting with 1. The version-control variable dictates how emacs saves backups.

Saving the buffer

The command CONTROL-X CONTROL-S saves the current buffer in its associated file. The emacs editor confirms a successful save with a message in the Echo Area.

Visiting another file

If you are already editing a file with emacs and want to edit another file (also called visiting a file), you can copy the new file into a new emacs buffer by giving the command CONTROL-X CONTROL-F. The emacs editor prompts you for a filename, reads that file into a new buffer, and displays that buffer in the current window. Having two files open in one editing session is more convenient than exiting from emacs, returning to the shell, and then starting a new copy of emacs to edit a second file.

Tip: Visiting a file with CONTROL-X CONTROL-F

When you use CONTROL-X CONTROL-F, emacs partially completes the path to the filename you are to enter. Normally it is the path to the working directory, but in some situations emacs may display a different path, such as the path to your home directory. You can edit this path if it is not pointing to the directory you want.


Basic Editing Commands

This section takes a more detailed look at the fundamental emacs editing commands. It covers straightforward editing of a single file in a single window.

Keys: Notation and Use

Mainstream emacs uses the 128-character ASCII character set. ASCII keyboards have a typewriter-style SHIFT key and a CONTROL key. Some keyboards also have a META (diamond, OPTION, or ALT) key that controls the eighth bit. It takes seven bits to describe an ASCII character; the eighth bit of an eight-bit byte can be used to communicate other information. Because so much of the emacs command set is in the nonprinting CONTROL or META case, Stallman was one of the first to confront the problem of developing a notation for writing about keystrokes.

His solution, although not popular outside the emacs community, is clear and unambiguous (Table 7-1). It uses the capital letters C and M to denote holding down the CONTROL and META (or ALT) keys, respectively, and a few simple acronyms for the most common special characters, such as RET (this book uses RETURN), LFD (LINEFEED), DEL (DELETE), ESC (ESCAPE), SPC (SPACE), and TAB. Most emacs documentation, including the online help, uses this notation.

Table 7-1. emacs key notation

Character

Classic emacs notation

(lowercase) a

a

(uppercase) SHIFT-a

A

CONTROL-a

C-a

CONTROL-A

C-a (do not use SHIFT), equivalent to CONTROL-a

META-a

M-a

META-A

M-A (do use SHIFT), different from M-a

CONTROL-META-a

C-M-a

META-CONTROL-a

M-C-a (not used frequently)

This use of keys had some problems. Many keyboards had no META key, and some operating systems discarded the META bit. In addition, the emacs character set clashes with XON-XOFF flow control, which also uses CONTROL-S and CONTROL-Q.

Although the flow-control problem still exists, the META key issue was resolved by making it an optional two-key sequence starting with ESCAPE. For instance, you can type ESCAPE-a instead of META-a or type ESCAPE CONTROL-A to get CONTROL-META-a. If the keyboard you are using does not have a META key, you can use the two-key ESCAPE sequence by pressing the ESCAPE key, releasing it, and then pressing the key following the META key in this book. For example, when this book says to press META-r, you can either press the META or ALT key while you press r or press and release ESCAPE and then press r.

Tip: Using emacs from the Terminal utility

By default the Terminal utility does not treat the standard Apple keyboard OPTION key as a META key. You must either use ESCAPE or configure Terminal to use OPTION as META. If you are using a non-Apple keyboard, the key may be labeled ALT, but you must still configure it in the same way. For more information see the tip "Activating the META key" on page 31.

Tip: The notation used in this book

This book uses an uppercase letter following the CONTROL key and a lowercase letter following the META key. In either case you do not have to hold down the SHIFT key while entering a CONTROL or META character. Although the META uppercase character (that is, META-A) is a different character, it is usually set up to cause no action or the same effect as its lowercase counterpart.

Key Sequences and Commands

In emacs the relationship between key sequences (one or more keys that are pressed together or in sequence to issue an emacs command) and commands is very flexible, and there is considerable opportunity for exercising your personal preference. You can translate and remap key sequences to other commands and replace or reprogram commands.

Although most emacs documentation glosses over the details and talks about keystrokes as though they were the commands, it is important to recognize that the underlying machinery remains separate from the key sequences and to understand that you can change the behavior of the key sequences and the commands. For more information refer to "Customizing emacs" on page 238.

META-x: Running a Command Without a Key Binding

The emacs keymaps (the tables, or vectors, that emacs uses to translate key sequences to commands [page 240]) are very crowded, and often it is not possible to bind every command to a key sequence. You can execute any command by name by preceding it with META-x. When you press META-x, the emacs editor prompts you for a command in the Echo Area. After you enter the command name and press RETURN, it executes the command.

Smart completion

When a command has no common key sequence, it is sometimes described as META-x command-name. The emacs editor has a smart completion for most prompted answers, using SPACE or TAB to complete, if possible, to the end of the current word or the whole command, respectively. Forcing a completion past the last unambiguous point or typing ? displays a list of alternatives. You can find more details on smart completion in the online emacs manual.

Numeric Arguments

Some of the emacs editing commands accept a numeric argument as a repetition count. You place this argument immediately before the key sequence for the command. Absence of an argument almost always means a count of 1. Even an ordinary alphabetic character can have a numeric argument, which means "insert this many times." To give a command a numeric argument, you can do either of the following:

  • Press META with each digit (09) or the minus sign (). For example, to insert 10 z characters, type META-1 META-0 z.

  • Use CONTROL-U to begin a string of digits, including the minus sign. For example, to move the cursor forward 20 words, type CONTROL-U 20 META-f.

CONTROL-U

For convenience, CONTROL-U defaults to multiply by 4 when you do not follow it with a string of one or more digits. For example, entering CONTROL-U r means insert rrrr (4 * 1), whereas CONTROL-U CONTROL-U r means insert rrrrrrrrrrrrrrrr (4 * 4 * 1). For quick partial scrolling of a tall window, you may find it convenient to use repeated sequences of CONTROL-U CONTROL-V to scroll down four lines, CONTROL-U META-v to scroll up four lines, CONTROL-U CONTROL-U CONTROL-V to scroll down 16 lines, or CONTROL-U CONTROL-U META-v to scroll up 16 lines.

Point and the Cursor

Point is the place in a buffer where editing takes place and is where the cursor is positioned. Strictly speaking, Point is the left edge of the cursorit is thought of as lying between two characters.

Each window has its own Point, but there is only one cursor. When the cursor is in a window, moving the cursor also moves Point. Switching the cursor out of a window does not change that window's Point; it is in the same place when you switch the cursor back to that window.

All of the cursor-movement commands described previously also move Point.

Scrolling Through a Buffer

CONTROL-V META-v CONTROL-L

A buffer is likely to be much larger than the window through which it is viewed, so you need a way of moving the display of the buffer contents up or down to position the interesting part in the window. Scrolling forward refers to moving the text upward, with new lines entering at the bottom of the window. Use CONTROL-V or the PAGE DOWN key to scroll forward one window (minus two lines for context). Scrolling backward refers to moving the text downward, with new lines entering at the top of the window. Use META-v or the PAGE UP key to scroll backward one window (again leaving two lines for context). Pressing CONTROL-L clears the screen and repaints it, moving the current line to the center of the window. This command is useful if the screen becomes garbled.

A numeric argument to CONTROL-V or META-v means "scroll that many lines"; thus CONTROL-U 10 CONTROL-V means scroll forward ten lines. A numeric argument to CONTROL-L means "scroll the text so the cursor is on that line of the window," where 0 means the top line and 1 means the bottom, just above the Mode Line. Scrolling occurs automatically if you exceed the window limits with CONTROL-P or CONTROL-N.

META-< META->

You can move the cursor to the beginning of the buffer with META-< or to the end of the buffer with META->.

Erasing Text

Delete versus kill

When you erase text you can discard it or move it into a holding area and optionally bring it back later. The term delete means permanently discard, and the term kill means move to a holding area. The holding area, called the Kill Ring, can hold several pieces of killed text. You can use the text in the Kill Ring in many ways (refer to "Cut and Paste: Yanking Killed Text" on page 218).

The META-d command kills from the cursor forward to the end of the current word. CONTROL-K kills forward to the end of the current line. It does not delete the line-ending LINEFEED character unless Point and the cursor are just to the left of the LINEFEED. This setup allows you to reach the left end of a line with CONTROL-A, kill the whole line with CONTROL-K, and then immediately type a replacement line without having to reopen a hole for the new line. Another consequence is that, from the beginning of the line, it takes CONTROL-K CONTROL-K (or CONTROL-U 2 CONTROL-K) to kill the text and close the hole.

Searching

The emacs editor has several types of search commands. You can search in the following ways:

  • Incrementally for a character string

  • Incrementally for a regular expression (possible but uncommon)

  • For a complete character string

  • For a complete regular expression (Appendix A)

You can run each of the four types of searches either forward or backward in the buffer.

The complete searches behave in the same manner as a search on other editors. Searching begins only when the search string is complete. In contrast, an incremental search begins when you type the first character of the search string and keeps going as you enter additional characters. Initially this approach may sound confusing, but it is surprisingly useful.

Incremental Searches

CONTROL-S CONTROL-R

A single command selects the direction of and starts an incremental search. CONTROL-S starts a forward incremental search, and CONTROL-R starts a reverse incremental search.

When you start an incremental search, emacs prompts you with I-search: in the Echo Area. When you enter a character, it immediately searches for that character in the buffer. If it finds that character, emacs moves Point and cursor to that position so you can see the search progress. If the search fails, emacs tells you so.

After you enter each character of the search string, you can take one of several actions depending on the result of the search to that point.

  • The search finds the string you are looking for in the buffer, leaving the cursor positioned just to its right. Stop the search and leave the cursor in its new position by pressing RETURN. (Any emacs command not related to searching will also stop the search but remembering exactly which ones apply can be difficult. For a new user, RETURN is safer.)

  • The search finds a string but it is not the one you are looking for. You can refine the search string by adding another letter, press CONTROL-R or CONTROL-S again to look for the next occurrence of this search string, or press RETURN to stop the search and leave the cursor where it is.

  • The search hits the beginning or end of the buffer and reports Failing I-Search. You can proceed in several ways at this point.

    • If you mistyped the search string, press BACKSPACE as needed to remove characters from the search string. The text and cursor in the window jump backward in step with your removal of characters.

    • If you want to wrap past the beginning or end of the buffer and continue searching, you can force a wrap by pressing CONTROL-R or CONTROL-S again.

    • If the search has not found the string you are looking for but you want to leave the cursor at its current position, press RETURN to stop the search.

    • If the search has gone wrong and you just want to get back to where you started, press CONTROL-G (the quit character). From an unsuccessful search a single CONTROL-G backs out all the characters in the search string that could not be found. If this action returns you to a place you wish to continue searching from, you can add characters to the search string again. If you do not want to continue the search from that position, a second CONTROL-G stops the search and leaves the cursor where it was initially.

Nonincremental Searches

CONTROL-S RETURN CONTROL-R RETURN

If you prefer that your searches succeed or fail without showing all the intermediate results, you can give the nonincremental command CONTROL-S RETURN to search forward or CONTROL-R RETURN to search backward. Searching does not begin until you enter a search string in response to the emacs prompt and press RETURN again. Neither of these commands wraps past the end of the buffer.

Regular Expression Searches

You can perform both incremental and nonincremental regular expression searching in emacs. Use the commands listed in Table 7-2 to begin a regular expression search.

Table 7-2. Searching for regular expressions

Command

Result

META-CONTROL-s

Incrementally searches forward for a regular expression; prompts for a regular expression one character at a time

META-x isearch-backward-regexp

Incrementally searches backward for a regular expression; prompts for a regular expression one character at a time

META-x isearch-complete RETURN

Prompts for and then searches forward for a complete regular expression

META-x isearch-backward-regexp RETURN

Prompts for and then searches backward for a complete regular expression


Online Help

The emacs help system is always available. With the default key bindings, you can start it with CONTROL-H. The help system then prompts you for a one-letter help command. If you do not know which help command you want, type ? or CONTROL-H to switch the current window to a list of help commands, each with a one-line description; emacs again requests a one-letter help command. If you decide you do not want help after all, type CONTROL-G to cancel your help request and return to the former buffer.

If the help output is only a single line, it appears in the Echo Area. If it consists of more text, the output appears in its own window. Use CONTROL-V and META-v to scroll forward and backward through the buffer (page 210). You can move the cursor between windows with CONTROL-X o (lowercase "o"). See page 225 for a discussion on working with multiple windows.

Tip: Closing the help window

To delete the help window while the cursor is in the window that holds the text you are editing, type CONTROL-X 1 (one). Alternatively, you can move the cursor to the help window (CONTROL-X 0 [lowercase "o"]) and type CONTROL-X 0 (zero) to delete the current window.

If help displays a window that occupies the entire screen, as is the case with CONTROL-H n (emacs news) and CONTROL-H t (emacs tutorial), you can kill the help buffer with CONTROL-X k or use CONTROL-X b to switch buffers (both on page 224).

On many terminals the BACKSPACE or LEFT ARROW key generates CONTROL-H. If you forget that you are using emacs and try to back over a few characters, you may unintentionally enter the help system. This action does not pose a danger to the buffer you are editing, but it can be unsettling to lose the window contents and not have a clear picture of how to restore it. While you are being prompted for the type of help you want you can type CONTROL-G to remove the prompt and return to editing the buffer. Some users elect to put help on a different key (page 240). Table 7-3 lists some of the help commands.

Table 7-3. Help commands

Command

Type of help offered

CONTROL-H a

Prompts for a string and displays a list of commands whose names contain that string.

CONTROL-H b

Displays a long table of the key bindings in effect.

CONTROL-H c key-sequence

Displays the name of the command bound to key-sequence. Multiple key sequences are allowed. For a long key sequence where only the first part is recognized, the command describes the first part and quietly inserts the unrecognized part into the buffer. This can happen with three-character function keys (F1, F2, and so on, on the keyboard) that generate character sequences such as ESCAPE [SHIFT.

CONTROL-H f

Prompts for the name of a Lisp function and displays the documentation for it. Because commands are Lisp functions, you can use a command name with this command.

CONTROL-H i

Displays the top info (page 30) menu where you can browse emacs or other documentation.

CONTROL-H k key-sequence

Displays the name and documentation of the command bound to key-sequence. (See the notes on CONTROL-H c.)

CONTROL-H l (lowercase "l")

Displays the last 100 characters typed. The record is kept after the first-stage keyboard translation. If you have customized the keyboard translation table, you must make a mental reverse translation.

CONTROL-H m

Displays the documentation and special key bindings for the current Major mode (Text, C, Fundamental, and so on, [page 229]).

CONTROL-H n

Displays the emacs news file which lists recent changes to emacs, ordered with the most recent changes first. See the tip "Closing the help window" on page 213.

CONTROL-H t

Runs an emacs tutorial session. See the tip "Closing the help window" on page 213.

CONTROL-H v

Prompts for a Lisp variable name and displays the documentation for that variable.

CONTROL-H w

Prompts for a command name and identifies any key sequence bound to that command. Multiple key sequences are allowed. (See the notes on CONTROL-H c.)

As this abridged presentation makes clear, you can use the help system to browse through the emacs internal Lisp system. For the curious following is Stallman's list of strings that match many names in the Lisp system. To get a view of the internal functionality of emacs, you can use any of these strings with CONTROL-H a (help system list of commands) or META-x apropos (prompts for a string and lists variables whose names contain that string).

backward | dir | insert | previous | view
beginning | down | kill | region | what
buffer | end | line | register | window
case | file | list | screen | word
change | fill | mark | search | yank
char | find | mode | sentence |
defun | forward | next | set |
delete | goto | page | sexp |
describe | indent | paragraph | up |


Advanced Editing

The basic emacs commands suffice for many editing tasks but the serious user will quickly discover the need for more power. This section presents some of the more advanced emacs capabilities.

Undoing Changes

An editing session begins when you read a file into an emacs buffer. At that point the buffer content matches the file exactly. As you insert text and give editing commands, the buffer content becomes increasingly more different from the file. If you are satisfied with the changes, you can write the altered buffer back out to the file and end the session.

Near the left end of the Mode Line (Figure 7-1, page 202) is an indicator that shows the modification state of the buffer that is displayed in the window. The three possible states are -- (not modified), * * (modified), and %% (readonly).

The emacs editor keeps a record of all the keys you have pressed (text and commands) since the beginning of the editing session, up to a limit currently set at 20,000 characters. If you are within this limit, it is possible to undo the entire session for this buffer, one change at a time. If you have multiple buffers (page 224), each buffer has its own undo record.

Undoing is considered so important that it has a backup key sequence, just in case some keyboards cannot easily handle the primary sequence. The two sequences are CONTROL-_ (underscore, which on old ASR-33 TTY keyboards was LEFT ARROW) and CONTROL-X u. When you type CONTROL-_, emacs undoes the last command and moves the cursor to that position in the buffer so you can see what happened. If you type CONTROL-_ a second time, the next-to-last command is undone, and so on. If you keep typing CONTROL-_, eventually you will get the buffer back to its original unmodified state and the * * Mode Line indicator will change to --.

When you break the string of Undo commands with anything (text or any command except Undo), all reverse changes you made during the string of undos become a part of the change record and can themselves be undone. This strategy offers a way to redo some or all the undo operations. If you decide you backed up too far, type a command (something innocuous, such as CONTROL-F, that does not change the buffer), and begin undoing in reverse. Table 7-4 lists some examples of Undo commands.

Table 7-4. Undo commands

Commands

Result

CONTROL-_

Undoes the last change

CONTROL-_CONTROL-F CONTROL-_

Undoes the last change and changes it back again

CONTROL-_CONTROL-_

Undoes the last two changes

CONTROL-_CONTROL-_ CONTROL-F CONTROL-_ CONTROL-_

Undoes two changes and changes them both back again

CONTROL-_CONTROL-_ CONTROL-F CONTROL-_

Undoes two changes and changes one of them back again

If you do not remember the last change you made, you can type CONTROL-_ and undo it. If you wanted to make this change, type CONTROL-F CONTROL-_ and make it again. If you modified a buffer by accident, you can keep typing CONTROL-_ until the Mode Line indicator shows once more.

If the buffer is completely ruined and you want to start over, issue the command META-x revert-buffer to discard the current buffer contents and reread the associated file. The emacs editor asks you to confirm this command.

Mark and Region

Point is the current editing position in a buffer which you can move anywhere within the buffer by moving the cursor. It is also possible to set a marker called Mark in the buffer. The contiguous characters between Point and Mark (either one may come first) are called the Region. Many commands operate on a buffer's Region, not just on the characters near Point.

Moving Mark and Establishing a Region

CONTROL-@ CONTROL-SPACE CONTROL-X CONTROL-X

Mark is not as easy to move as Point. Once set, Mark can be moved only by setting it somewhere else. Each buffer has only one Mark. The CONTROL-@ (or CONTROL-SPACE) command explicitly sets Mark at the current cursor (and Point) position. Some key-boards generate CONTROL-@ when you type CONTROL-Q. Although this is not really a backup key binding, it is occasionally a convenient alternative. You can use CONTROL-X CONTROL-X to exchange Point and Mark (and move the cursor to the new Point).

To establish a Region, you usually position the cursor (and Point) at one end of the desired Region, set Mark with CONTROL-@, and then move the cursor (and Point) to the other end of the Region. If you forget where you left Mark, you can move the cursor back to it again with CONTROL-X CONTROL-X or hop back and forth with repeated CONTROL-X CONTROL-X to show the Region more clearly.

If a Region boundary is not to your liking, you can swap Point and Mark using CONTROL-X CONTROL-X to move the cursor from one end of the Region to the other and then move Point. Continue until you are satisfied with the Region.

Operating on a Region

Table 7-5 lists selected commands that operate on a Region. Give the command CONTROL-H a region to see a complete list of these commands.

Table 7-5. Operating on a region

Command

Result

META-w

Copies the Region nondestructively (without killing it) to the Kill Ring

CONTROL-W

Kills the Region

META-x print-region

Sends the Region to the printer

META-x append-to-buffer

Prompts for a buffer and appends the Region to that buffer

META-x append-to-file

Prompts for a filename and appends the Region to that file

META-x capitalize-region

Converts the Region to uppercase

The Mark Ring

Each time you set Mark in a buffer, you are also pushing Mark's former location onto the buffer's Mark Ring. The Mark Ring is organized as a FIFO (first-in-first-out) list and holds the 16 most recent locations where Mark was set. Each buffer has its own Mark Ring. This record of recent Mark history is useful because it often holds locations that you want to jump back to quickly. Jumping to a location pointed to by the Mark Ring can be faster and easier than scrolling or searching your way through the buffer to find the site of a previous change.

CONTROL-U CONTROL-@

To work your way backward along the trail of former Mark locations, give the command CONTROL-U CONTROL-@ one or more times. Each time you give the command, emacs

  • Moves Point (and the cursor) to the current Mark location

  • Saves the current Mark location at the oldest end of the Mark Ring

  • Pops off the youngest (most recent) Mark Ring entry and sets Mark

Each additional CONTROL-U CONTROL-@ command causes emacs to move Point and the cursor to the previous entry on the Mark Ring.

Although this process may seem complex, it really just makes a safe jump to a previous Mark location. It is safe because each jump's starting point is recirculated through the Mark Ring, where it is easy to find again. You can jump to all previous locations on the Mark Ring (it may be fewer than 16) by giving the command CONTROL-U CONTROL-@ again and again. You can go around the ring as many times as you like and stop whenever you want.

Setting Mark Automatically

Some commands set Mark automatically: The idea is to leave a bookmark before moving Point a long distance. For example, META-> sets Mark before jumping to the end of the buffer. You can then return to your starting position with CONTROL-U CONTROL-@. Searches behave similarly. To avoid surprises the message Mark Set appears in the Echo Area whenever Mark is set, either explicitly or implicitly.

Cut and Paste: Yanking Killed Text

Recall that killed text is not discarded but rather is kept in the Kill Ring. The Kill Ring holds the last 30 pieces of killed text and is visible from all buffers.

Retrieving text from the Kill Ring is called yanking. This terminology is the opposite of that used in vim: In vim yanking pulls text from the buffer, and putting puts text into the buffer. Killing and yankingwhich are roughly analogous to cutting and pastingare emacs's primary mechanisms for moving and copying text. Table 7-6 lists the most common kill and yank commands.

Table 7-6. Common kill and yank commands

Command

Result

META-d

Kills to end of current word

META-D

Kills from beginning of previous word

CONTROL-K

Kills to end of line, not including LINEFEED

CONTROL-U 1 CONTROL-K

Kills to end of line, including LINEFEED

CONTROL-U 0 CONTROL-K

Kills from beginning of line

META-w

Copies the Region (between Point and Mark) to the Kill Ring but does not erase the Region from the buffer

CONTROL-W

Kills the Region (between Point and Mark)

META-z char

Kills up to next occurrence of char

CONTROL-Y

Yanks the most recently killed text into the current buffer at Point, sets Mark at the beginning of this text, and positions Point and the cursor at the end

META-y

Erases the just-yanked text, rotates the Kill Ring, and yanks the next item (only after CONTROL-Y or META-y)

To move two lines of text, move Point to the beginning of the first line and then enter CONTROL-U 2 CONTROL-K to kill two lines. Next move Point to the destination position, and enter CONTROL-Y.

To copy two lines of text, move Point to the beginning of the first line and give the commands CONTROL-U 2 CONTROL-K CONTROL-Y to kill and then yank back immediately. Then move Point to the destination position and type CONTROL-Y.

To copy a larger piece of the buffer, set the Region to cover this piece and then type CONTROL-W CONTROL-Y to kill and yank back at once. Next move Point to the destination, and type CONTROL-Y. You can also set the Region and use META-w to copy the Region to the Kill Ring.

The Kill Ring is organized as a fixed-length FIFO list, with each new entry causing the eldest to be discarded (once you build up to 30 entries). Simple cut-and-paste operations generally use only the newest entry. The older entries are retained to give you time to change your mind about a deletion. If you do change your mind you can "mine" the Kill Ring like an archaeological dig, working backward through time and down through the strata of killed material to copy a specific item back into the buffer.

To view every entry in the Kill Ring, begin a yanking session by pressing CONTROL-Y. This action copies the youngest entry to your buffer at the current cursor position. If this entry is not the item you want, continue the yanking session by pressing META-y. This action erases the previous yank and copies the next youngest entry to the buffer at the current cursor position. If this still is not the item you wanted, press META-y again to erase it and retrieve a copy of the next entry, and so on. You can continue giving META-y commands all the way back to the oldest entry. If you continue to press META-y, you wrap back to the youngest entry again. In this manner you can examine each entry as many times as you wish.

The sequence used in a yanking session consists of CONTROL-Y followed by any mixture of CONTROL-Y and META-y. If you type any other command after META-y, the sequence is broken and you must give the CONTROL-Y command again to start another yanking session.

As you work backward in the Kill Ring, it is useful to think of this process as advancing a Last Yank pointer back through history to increasingly older entries. This pointer is not reset to the youngest entry until you give a new kill command. Using this technique, you can work backward partway through the Kill Ring with CONTROL-Y and a few META-y commands, give some commands that do not kill, and then pick up where you left off with another CONTROL-Y and a succession of META-y commands.

It is also possible to position the Last Yank pointer with positive or negative numeric arguments to META-y. Refer to the online documentation for more information.

Inserting Special Characters

As stated earlier, emacs inserts everything that is not a command into the buffer at the position of the cursor. To insert characters that would ordinarily be emacs commands, you can use the emacs escape character: CONTROL-Q. There are two ways of using this escape character:

  • CONTROL-Q followed by any other character inserts that character in the buffer, no matter what command interpretation it was supposed to have.

  • CONTROL-Q followed by three octal digits inserts a byte with that value in the buffer.

Tip: CONTROL-Q

Depending on the way your terminal is set up, CONTROL-Q may clash with software flow control. If CONTROL-Q seems to have no effect, it is most likely being used for flow control. In that case you must bind another key to the command quoted-insert (page 240).

Global Buffer Commands

The vim editor and its predecessors have global commands for bufferwide search and replace operations. Their default operating Region is the entire buffer. The emacs editor has a similar family of commands. Their operating Region begins at Point and extends to the end of the buffer. If you wish to operate on the entire buffer, use META-< to set Point at the beginning of the buffer before issuing the command.

Line-Oriented Operations

The commands listed in Table 7-7 take a regular expression and apply it to the lines between Point and the end of the buffer.

Table 7-7. Line-oriented operations

Command

Result

META-x occur

Prompts for a regular expression and copies each line with a match for the expression in a buffer named *Occur*

META-x delete-matching-lines

Prompts for a regular expression and deletes each line with a match for the expression

META-x delete-non-matching-lines

Prompts for a regular expression and deletes each line that does not have a match for that expression

The META-x occur command puts its output in a special buffer named *Occur*, which you can peruse and discard or use as a jump menu to reach each line quickly. To use the *Occur* buffer as a jump menu, switch to it (CONTROL-X o [lowercase "o"]), move the cursor to the copy of the desired destination line, and type CONTROL-C CONTROL-C. This command moves the cursor to the buffer that was searched and positions it on the line that the regular expression matched.

As with any buffer change, you can undo the deletion commands.

Unconditional and Interactive Replacement

The commands listed in Table 7-8 operate on the characters between Point and the end of the buffer, changing every string match or regular expression match. An unconditional replacement makes all replacements automatically. An interactive replacement gives you the opportunity to see and approve each replacement before it is made.

Table 7-8. Replacement commands

Command

Result

META-x replace-string

Prompts for string and newstring and replaces every instance of string with newstring. Point is left at the site of the last replacement, but Mark is set when you give the command, so you can return to it with CONTROL-U CONTROL-@.

META-x replace-regexp

Prompts for regexp and newstring and replaces every match for regexp with newstring. Point is left at the site of the last replacement, but Mark is set when you give the command, so you can return to it with CONTROL-U CONTROL-@.

META-% string or META-x query-replace

The first form uses string, the second form prompts for string. Both forms prompt for newstring, query each instance of string, and, depending on your response, replace it with newstring. Point is left at the site of the last replacement, but Mark is set when you give the command, so you can return to it with CONTROL-U CONTROL-@.

META-x query-replace-regexp

Prompts for regexp and newstring, queries each match for regexp, and, depending on your response, replaces it with newstring. Point is left at the site of the last replacement, but Mark is set when you give the command, so you can return to it with CONTROL-U CONTROL-@.

If you perform an interactive replacement, emacs displays each instance of string or match for regexp and prompts you for an action to take. Table 7-9 lists some of the possible responses.

Table 7-9. Responses to interactive replacement prompts

Response

Meaning

RETURN

Do not do any more replacements; quit now.

SPACE

Make this replacement and go on.

DELETE

Do not make this replacement. Skip it and go on.

, (comma)

Make this replacement, display the result, and ask for another command. Any command is legal except DELETE is treated like SPACE and does not undo the change.

. (period)

Make this replacement and quit searching.

! (exclamation point)

Replace this and all remaining instances without asking any more questions.

Files

When you visit (emacs terminology for "call up") a file, emacs reads it into a buffer (page 224), allows you to edit the buffer, and eventually saves the buffer back to the file. The commands discussed here relate to visiting and saving files.

META-x pwd META-x cd

Each emacs buffer keeps a record of its default directory (the directory the file was read from or the working directory, if it is a new file) that is prepended to any relative pathname you specify. This convenience is meant to save some typing. Enter META-x pwd to print the default directory for the current buffer or META-x cd to prompt for a new default directory and assign it to this buffer.

Visiting Files

The emacs editor deals well with visiting a file that has already been called up and whose image is now in a buffer. After a check of the modification time to ensure that the file has not been changed since it was last called up, emacs simply switches to that buffer. Table 7-10 lists commands used to visit files.

Table 7-10. Visiting files

Command

Result

CONTROL-X CONTROL-F

Prompts for a filename and reads its contents into a freshly created buffer. Assigns the file's simple filename as the buffer name. Other buffers are unaffected. It is common and often useful to have several files open simultaneously for editing.

CONTROL-X CONTROL-V

Prompts for a filename and replaces the current buffer with a buffer containing the contents of the requested file. The current buffer is destroyed.

CONTROL-X 4 CONTROL-F

Prompts for a filename and reads its contents into a new buffer. Assigns the file's simple filename as the buffer name. Creates a new window for this buffer and selects that window. The window selected before the command still displays the buffer it was showing before this operation, although the new window may cover up part of the old window.

To create a new file, simply call it up. An empty buffer is created and properly named so you can eventually save it. The message (New File) appears in the Echo Area, reflecting emacs's understanding of the situation. Of course, if this "new file" grew out of a typographical error, you will probably want to issue CONTROL-X CONTROL-V with the correct name.

Saving Files

You save a buffer by copying its contents back to the original file you called up. The relevant commands are listed in Table 7-11.

Table 7-11. Saving files

Command

Result

CONTROL-X CONTROL-S

This workhorse file-saving command saves the current buffer into its original file. If the current buffer is not modified, you get the following message: (No changes need to be saved).

CONTROL-X s

For each modified buffer, you are asked whether you wish to save it. Answer y or n. This command is given automatically as you exit from emacs and allows you to save any buffers that have been modified but not yet written out. If you want to save intermediate copies of your work, you can give this command at any time.

META-x set-visited-file-name

Prompts for a filename and sets this name as the current buffer's "original" name.

CONTROL-X CONTROL-W

Prompts for a filename, sets this name as the "original" name for the current buffer, and saves the current buffer into that file. It is equivalent to META-x set-visited-file-name followed by CONTROL-X CONTROL-S.

META-~ (tilde)

Clears modified flag from the current buffer. If you mistakenly typed META-~ against a buffer with changes you want to keep, you need to make sure that the modified condition and its * * indicator are turned back on before leaving emacs, or all the changes will be lost. One easy way to do this is to insert a SPACE into the buffer and then remove it again with DELETE.

Caution: Did you modify a buffer by mistake?

When you give a CONTROL-X s command, you may discover files whose buffers were modified by mistake as emacs TRies to save the wrong changes back to the file. When emacs prompts you to confirm the save, do not answer y if you are not sure. First exit from the CONTROL-X s dialog by typing n to any saves you are not sure about. You then have several options:

  • Save the suspicious buffer into a temporary file with CONTROL-X CONTROL-W and analyze it later.

  • Undo the changes with a string of CONTROL-_ commands until the * * indicator disappears from the buffer's Mode Line.

  • If you are sure that all the changes are wrong, use META-x revert-buffer to get a fresh copy of the file.

  • Kill the buffer outright. Because it is modified, emacs asks whether you are sure before carrying out this command.

  • Give the META-~ (tilde) command to clear the modified condition and * * indicator. A subsequent CONTROL-X s then believes that the buffer does not need to be written.

Caution: You can exit without first getting a warning

Clearing the modified flag (META-~) allows you to exit without saving a modified buffer with no warning. Make sure you know what you are doing when you use META-~.

Buffers

An emacs buffer is a storage object that you can edit. It often holds the contents of a file but can also exist without being associated with a file. You can select only one buffer at a time, designated as the current buffer. Most commands operate only on the current buffer, even when multiple windows show two or more buffers on the screen. For the most part each buffer is its own world: It has its own name, its own modes, its own file associations, its own modified state, and perhaps its own special key bindings. You can use the commands shown in Table 7-12 to create, select, list, and manipulate buffers.

Table 7-12. Work with buffers

Command

Result

CONTROL-X b

Prompts for a buffer name and selects it. If the buffer you name does not exist, this command creates it.

CONTROL-X 4 b

Prompts for a buffer name and selects it in another window. The existing window is not disturbed, although the new window may overlap it.

CONTROL-X CONTROL-B

Creates a buffer named *Buffer List* and displays it in another window. The existing window is not disturbed, although the new window may overlap it. The new buffer is not selected. In the *Buffer List* buffer, each buffer's data is shown along with the name, size, mode(s), and original filename. A % appears for a readonly buffer, a * indicates a modified buffer, and . appears for the selected buffer.

META-x rename-buffer

Prompts for a new buffer name and gives this new name to the current buffer.

CONTROL-X CONTROL-Q

Toggles the current buffer's readonly status and the associated %% Mode Line indicator. This can be useful to prevent accidental buffer modification or to allow modification of a buffer when visiting a readonly file.

META-x append-to-buffer

Prompts for a buffer name and appends the Region (between Point and Mark) to the end of that buffer.

META-x prepend-to-buffer

Prompts for a buffer name and prepends the Region (between Point and Mark) to the beginning of that buffer.

META-x copy-to-buffer

Prompts for a buffer name and deletes the contents of the buffer before copying the Region (between Point and Mark) to that buffer.

META-x insert-buffer

Prompts for a buffer name and inserts the entire contents of that buffer into the current buffer at Point.

CONTROL-X k

Prompts for a buffer name and deletes that buffer. If the buffer is modified but unsaved, you are asked to confirm the operation.

META-x kill-some-buffers

Goes through the entire buffer list and offers the chance to delete each buffer. As with CONTROL-X k, you are asked to confirm the kill command if a modified buffer is not yet saved.

Windows

An emacs window is a viewport that looks into a buffer. The emacs screen begins by displaying a single window, but this screen space can later be divided among two or more windows. On the screen the current window holds the cursor and views the current buffer. For a tip on terminology, see "The screen and emacs windows" on page 201.

A window views one buffer at a time. You can switch the buffer that a window views by giving the command CONTROL-X b buffer-name in the current window. Multiple windows can view one buffer; each window may view different parts of the same buffer; and each window has its own Point value. Any change to a buffer is reflected in all the windows viewing that buffer. Also, a buffer can exist without a window open on it.

Splitting a Window

One way to divide the screen is to split the starting window explicitly into two or more pieces. The command CONTROL-X 2 splits the current window in two, with one new window appearing above the other. A numeric argument is taken as the size of the upper window in lines. The command CONTROL-X 3 splits the current window in two, with the new windows being arranged side by side (Figure 7-7). A numeric argument is taken as the number of columns to give the left window. For example, CONTROL-U CONTROL-X 2 splits the current window in two; because of the special "times 4" interpretation of CONTROL-U standing alone, the upper window is given four lines (barely enough to be useful).

Figure 7-7. Splitting a window vertically

[View full size image]

Although these commands split the current window, both windows continue to view the same buffer. You can select a new buffer in either or both new windows, or you can scroll each window to show different portions of the same buffer.

Manipulating Windows

CONTROL-X 0 META-CONTROL-V

You can use CONTROL-X o (lowercase "o") to select the other window. If more than two windows appear on the screen, a sequence of CONTROL-X o commands cycles through them in top-to-bottom, left-to-right order. The META-CONTROL-V command scrolls the other window. If more than two windows are visible, the command scrolls the window that CONTROL-X o would select next. You can use a positive or negative scrolling argument, just as with CONTROL-V scrolling in the current window.

Other-Window Display

CONTROL-X 4b CONTROL-X 4f

In normal emacs operation, explicit window splitting is not nearly as common as the implicit splitting done by the family of CONTROL-X 4 commands. The CONTROL-X 4b command, for example, prompts for a buffer name and selects it in the other window. If there is no other window, this command begins with a half-and-half split that arranges the windows one above the other. The CONTROL-X 4f command prompts for a filename, calls it up in the other window, and selects the other window. If there is no other window, this command begins with a half-and-half split that arranges the windows one above the other.

Adjusting and Deleting Windows

CONTROL-X 0 CONTROL-X 1

Windows may be destroyed when they get in the way. No data is lost in the window's associated buffer with this operation, and you can make another window whenever you like. The CONTROL-X 0 (zero) command deletes the current window and gives its space to its neighbors; CONTROL-X 1 deletes all windows except the current window.

META-x shrink-window CONTROL-X^ CONTROL-X} CONTROL-X{

You can also adjust the dimensions of the current window, once again at the expense of its neighbors. To make a window shorter, use META-x shrink-window. Use CONTROL-X^ to increase the height of a window, CONTROL-X} to make the window wider, and CONTROL-X{ to make the window narrower. Each of these commands adds or subtracts one line or column to or from the window, unless you precede the command with a numeric argument.

The emacs editor has its own guidelines for a window's minimum useful size and may destroy a window before you force one of its dimensions to zero. Although the window may disappear, the buffer remains intact.

Foreground Shell Commands

The emacs editor can run a subshell (a shell that is a child of the shell that is running emacsrefer to "Executing a Command" on page 294) to execute a single command line, optionally with standard input coming from the Region of the current buffer and optionally with standard output replacing the Region (Table 7-13). This process is analogous to executing a shell command from the vim editor and having the input come from the file you are editing and the output go back to the same file (page 187). As with vim, how well this process works depends in part on the capabilities of the shell.

Table 7-13. Foreground shell commands

Command

Result

META-! (exclamation point)

Prompts for a shell command, executes it, and displays the output

CONTROL-U META-! (exclamation point)

Prompts for a shell command, executes it, and inserts the output at Point

META-| (vertical bar)

Prompts for a shell command, gives the Region as input, filters it through the command, and displays the output

CONTROL-U META-| (vertical bar)

Prompts for a shell command, gives the Region as input, filters it through the command, deletes the old Region, and inserts the output in that position

The emacs editor can also start an interactive subshell that runs continuously in its own buffer. See "Shell Mode" on page 237 for more information.

Background Shell Commands

The emacs editor can run processes in the background, with their output being fed into a growing emacs buffer that does not have to remain in view. You can continue editing while the background process runs and look at its output later. Any shell command can be run in this way.

The growing output buffer is always named *compilation*. You can read it, copy from it, or edit it in any way, without waiting for the background process to finish. Most commonly this buffer is used to review the output of program compilation and to correct any syntax errors found by the compiler.

META-x compile

To run a process in the background, give the command META-x compile to prompt for a shell command and begin executing it as a background process. The screen splits in half to show the *compilation* buffer.

You can switch to the *compilation* buffer and watch the execution, if you wish. To make the display scroll as you watch, position the cursor at the very end of the text with a META-> command. If you are not interested in this display just remove the window with CONTROL-X 0 (zero) if you are in it or CONTROL-X 1 otherwise and keep working. You can switch back to the *compilation* buffer later with CONTROL-X b.

You can kill the background process with META-x kill-compilation. The emacs editor asks for confirmation and then kills the background process.

If standard format error messages appear in *compilation*, you can automatically visit the line in the file where each error occurred. Give the command CONTROL-X' (back tick) to split the screen into two windows and visit the file and line of the next error message. Scroll the *compilation* buffer until this error message appears at the top of its window. Use CONTROL-U CONTROL-X' to start over with the first error message and visit that file and line.


Language-Sensitive Editing

The emacs editor has a large collection of feature sets, each specific to a certain variety of text. The feature sets are called Major modes. A buffer can have only one Major mode at any time.

A buffer's Major mode is private to the buffer and does not affect editing in any other buffer. If you switch to a new buffer having a different mode, rules for the new mode immediately take effect. To avoid confusion, the name of a buffer's Major mode appears in the Mode Line of any window viewing that buffer (Figure 7-1 on page 202).

The three classes of Major modes are used for the following tasks:

  • Editing human languages (for example, text, nroff, TeX)

  • Editing programming languages (for example, C, Fortran, Lisp)

  • Special purposes (for example, shell, mail, dired, ftp)

In addition, one Major modeFundamentaldoes nothing special. A Major mode usually sets up the following:

  • Special commands unique to the mode, possibly with their own key bindings. Languages may have just a few special commands, but special-purpose modes may have dozens.

  • Mode-specific character syntax and regular expressions defining word constituent characters, delimiters, comments, whitespace, and so on. This setup conditions the behavior of commands oriented to syntactic units, such as words, sentences, comments, or parenthesized expressions.

Selecting a Major Mode

META-x modename

The emacs editor chooses and sets a mode when a file is called up by matching the filename against a set of regular expression patterns describing the filename and filename extension. The explicit command to enter a Major mode is META-x modename. This command is rarely used except to correct wrong guesses.

A file can define its own mode by including the text * modename * somewhere in the first nonblank line, possibly inside a comment suitable for that programming language.

Human-Language Modes

A human language is meant eventually to be used by humans, possibly after being formatted by some text-formatting program. Human languages share many conventions about the structure of words, sentences, and paragraphs. With regard to these textual units, the major human language modes all behave in the same way.

Beyond this area of commonality, each mode offers additional functionality oriented to a specific text formatter, such as TeX, LaTeX, or nroff. Text-formatter extensions are beyond the scope of this presentation; the focus here is on the commands relating to human textual units (for example, words, sentences, and paragraphs).

Words

As a mnemonic aid, the bindings for words are defined parallel to the character-oriented bindings CONTROL-F, CONTROL-B, CONTROL-D, DELETE, and CONTROL-T.

Just as CONTROL-F and CONTROL-B move forward and backward over characters, META-f and META-b move forward and backward over words. They may start from a position inside or outside the word to be traversed, but in all cases Point finishes just beyond the word, adjacent to the last character skipped over. Both commands accept a numeric argument specifying the number of words to be traversed.

Just as CONTROL-D and DELETE delete characters forward and backward, the keys META-d and META-DELETE kill words forward and backward. They leave Point in exactly the same finishing position as META-f and META-b do, but they kill the words they pass over. They also accept a numeric argument.

META-t transposes the word before Point with the word after Point.

Sentences

As a mnemonic aid, three of the bindings for sentences are defined parallel to the line-oriented bindings: CONTROL-A, CONTROL-E, and CONTROL-K. The META-a command moves backward to the beginning of a sentence, and META-e moves forward to the end of a sentence. In addition, CONTROL-X DELETE kills backward to the beginning of a sentence; META-k kills forward to the end of a sentence.

The emacs editor recognizes the ends of sentences by referring to a regular expression that is kept in a variable named sentence-end. (If you are curious, give the command CONTROL-H v sentence-end RETURN to view this variable.) Briefly, it looks for the characters ., ?, or ! followed by two SPACEs or an end-of-line marker, possibly with close quotation marks or close braces.

The META-a and META-e commands leave Point adjacent to the first or last nonblank character in the sentence. They accept a numeric argument specifying the number of sentences to traverse; a negative argument runs them in reverse.

The META-k and CONTROL-X DELETE commands kill sentences forward and backward, in a manner analogous to CONTROL-K line kill. They leave Point in exactly the same finishing position as META-a and META-e do, but they kill the sentences they pass over. They also accept a numeric argument. CONTROL-X DELETE is useful for quickly backing out of a half-finished sentence.

Paragraphs

The META-{ command moves backward to the most recent paragraph beginning, and META-} moves forward to the next paragraph ending. The META-h command marks the paragraph (that is, puts Point at the beginning and Mark at the end) that the cursor is currently on, or the next paragraph if it is between paragraphs.

The META-} and META-{ commands leave Point at the beginning of a line, adjacent to the first character or last character of the paragraph. They accept a numeric argument specifying the number of paragraphs to traverse and run in reverse if given a negative argument.

In human-language modes, paragraphs are separated by blank lines and text-formatter command lines, and an indented line starts a paragraph. Recognition is based on the regular expressions stored in the variables paragraph-separate and paragraph-start. A paragraph is composed of complete lines, including the final line terminator. If a paragraph starts following one or more blank lines, the last blank line before the paragraph belongs to the paragraph.

Fill

The emacs editor can fill a paragraph to fit a specified width, breaking lines and rearranging them as necessary. Breaking takes place only between words and no hyphenation occurs. Filling can be done automatically as you type or in response to an explicit command.

The META-x auto-fill-mode command toggles Auto Fill mode on and off. When Auto Fill mode is on, emacs automatically breaks lines when you press SPACE or RETURN and are currently beyond the specified line width. This feature is useful when you are entering new text.

Auto Fill mode does not automatically refill the entire paragraph you are currently working on. If you add new text in the middle of a paragraph, Auto Fill mode breaks your new text as you type but does not refill the complete paragraph. To refill a complete paragraph or Region of paragraphs, use either META-q to refill the current paragraph or META-x fill-region to refill each paragraph in the Region between Point and Mark.

You can change the filling width from its default value of 70 by setting the fill-column variable. Use CONTROL-X f to set fill-column to the current cursor position and CONTROL-U nnn CONTROL-X f to set fill-column to nnn, where 0 is the left margin.

Case Conversion

The emacs editor can force words or Regions to all uppercase, all lowercase, or initial caps (the first letter of each word uppercase, the rest lowercase). Refer to Table 7-14.

Table 7-14. Case conversion

Command

Result

META-l (lowercase "l")

Converts word to the right of Point to lowercase

META-u

Converts word to the right of Point to uppercase

META-c

Converts word to the right of Point to initial caps

CONTROL-X CONTROL-L

Converts the Region (between Point and Mark) to lowercase

CONTROL-X CONTROL-U

Converts the Region (between Point and Mark) to uppercase

The word-oriented conversions move Point over the word just converted (just as META-f does), allowing you to walk through text and convert each word with META-l, META-u, or META-c, or skip over words to be left alone with META-f. A positive numeric argument converts that number of words to the right of Point, moving Point as it goes. A negative numeric argument converts that number of words to the left of Point but leaves Point stationary. This feature is useful for quickly changing the case of words you have just typed. Table 7-15 shows some examples.

Table 7-15. Examples of case conversion

Characters and commands

Result

HELLO META-META-l (lowercase "l")

hello

hello META-META-u

HELLO

hello META-META-c

Hello

The word conversions are not picky about beginning in the middle of a word. In all cases, they consider the first word-constituent character to the right of Point as the beginning of the word to be converted.

Text Mode

With very few exceptions, the commands for human-language text units, such as words and sentences, are always turned on and available, even in the programming-language modes. Text mode adds very little to these basic commands but is still worth turning on just to get the TAB key. Use the command META-x text-mode to activate Text mode.

In Text mode TAB runs the function tab-to-tab-stop. By default TAB stops are set every eight columns. You can adjust them with META-x edit-tab-stops, which switches to a special *Tab Stops* buffer. The current stops are laid out in this buffer on a scale for you to edit. The new stops are installed when or if you type CONTROL-C CONTROL-C. Of course, you are free to kill this buffer (CONTROL-X k) or switch away from it (CONTROL-X b) without ever changing the stops.

The tab stops you set with the META-x edit-tab-stops command affect only the interpretation of TAB characters arriving from the keyboard. The emacs editor automatically inserts enough spaces to reach the TAB stop. This command does not affect the interpretation of TAB characters already in the buffer or the underlying file. If you edit the TAB stops and then use them, you can still print the file and the hard copy will look the same as the text on the screen.

C Mode

Programming languages are read by humans but are interpreted by machines. Besides continuing to handle some of the human-language text units (for example, words and sentences), the major programming-language modes address several additional problems:

  • Balanced expressions enclosed by parentheses, brackets, or braces as textual units

  • Comments as textual units

  • Indention

The emacs editor includes Major modes to support C, Fortran, and several variants of Lisp. In addition, many users have contributed modes for their favorite languages. In these modes the commands for human textual units are still available, with occasional redefinitions. For example, a paragraph is bounded only by blank lines and indention does not signal a paragraph start. In addition, each mode has custom coding to handle the language-specific conventions for balanced expressions, comments, and indention. This presentation discusses only C mode.

Expressions

The emacs Major modes are limited to lexical analysis. They can recognize most tokens (for example, symbols, strings, and numbers) and all matched sets of parentheses, brackets, and braces. This is enough for Lisp but not for C. The C mode lacks a full-function syntax analyzer and is not prepared to recognize all of C's possible expressions.[1]

[1] In the emacs documentation the recurring term sexp refers to the Lisp term S-expression. Unfortunately, it is sometimes used interchangeably with expression, even though the language might not be Lisp.

Table 7-16 lists the emacs commands applicable to parenthesized expressions and some tokens. By design the bindings run parallel to the CONTROL commands for characters and the META commands for words. All these commands accept a numeric argument and run in reverse if that argument is negative.

Table 7-16. Commands for expressions and tokens

Command

Result

CONTROL-META-f

Moves forward over an expression. The exact behavior depends on which character lies to the right of Point (or left of Point, depending on which direction you are moving Point).

  • If the first nonwhitespace is an opening delimiter (parenthesis, bracket, or brace), Point is moved just past the matching closing delimiter.

  • If the first nonwhitespace is a token, Point is moved just past the end of this token.

CONTROL-META-b

Moves backward over an expression.

CONTROL-META-k

Kills an expression forward. This command leaves Point at the same finishing position as CONTROL-META-f but kills the expression it traverses.

CONTROL-META-@

Sets Mark at the position CONTROL-META-f would move to but does not change Point. To see the marked Region clearly, give a pair of CONTROL-X CONTROL-X commands to interchange Point and Mark.

Function Definitions

In emacs a balanced expression at the outermost level is considered to be a function definition and is often called a defun, even though that term is specific to Lisp. More generally it is understood to be a function definition in the language at hand.

In C mode a function definition includes the return data type, the function name, and the argument declarations appearing before the { character. Table 7-17 shows the commands for operating on function definitions.

Table 7-17. Function definitions

Command

Result

CONTROL-META-a

Moves to the beginning of the most recent function definition. Use this command to scan backward through a buffer one function at a time.

CONTROL-META-e

Moves to the end of the next function definition. Use this command to scan forward through a buffer one function at a time.

CONTROL-META-h

Puts Point at the beginning and Mark at the end of the current function definition (or next function definition, if between two). This command sets up an entire function definition for a Region-oriented operation such as kill.

Caution: Function indention style

The emacs editor assumes that an opening brace at the left margin is part of a function definition. This heuristic speeds up the reverse scan for a definition's leading edge. If your code has an indention style that puts that opening brace elsewhere, you may get unexpected results.

Indention

The emacs C mode has extensive logic to control the indention of C programs. Furthermore, you can adjust this logic for many different styles of C indention (Table 7-18).

Table 7-18. Indention commands

Command

Result

TAB

Adjusts the indention of the current line. TAB inserts or deletes whitespace at the beginning of the line until the indention conforms to the current context and rules in effect. Point is not moved unless it lies in the whitespace area; in that case it is moved to the end of the whitespace. TAB does not insert anything except leading whitespace, so you can hit it at any time and at any position in the line. If you really want to insert a tab in the text, use META-i or CONTROL-Q TAB.

LINEFEED

Shorthand for RETURN followed by TAB. The LINEFEED key is a convenience for entering new code, giving you an autoindent as you begin each line.

The next two commands indent multiple lines with a single command.

CONTROL-META-q

Reindents all lines inside the next pair of matched braces. CONTROL-META-q assumes that the left brace is correctly indented and drives the indention from there. If you need to adjust the left brace type TAB to the left of the brace before giving this command. All lines up to the matching brace are indented as if you had typed TAB on each one.

CONTROL-META-\

Reindents all lines in the Region (between Point and Mark). Put Point just to the left of a left brace and then give the command. All lines up to the matching brace are indented as if you had typed TAB on each one.

Customizing Indention

Many styles of C programming have evolved, and emacs does its best to support automatic indention for all of them. The indention coding was completely rewritten for emacs version 19; it supports C, C++, Objective-C, and Java. The new emacs syntactic analysis is much more precise and can classify each syntactic element of each line of program text into a single syntactic category (out of about 50), such as statement, string, or else-clause. With the result of that analysis in hand, emacs goes to an offset table named c-offsets-alist and looks up how much each line should be indented from the preceding line.

To customize indention, you must change the offset table. It is possible to define a completely new offset table for each customized style but much more convenient to feed in a short list of exceptions to the standard rules. Each mainstream style (GNU, K&R [Kernighan and Ritchie], BSD, and so on) has such an exception list; all are collected in c-style-alist. Here is one entry from c-style-alist:

("gnu"
(c-basic-offset . 2)
(c-comment-only-line-offset . (0 . 0))
(c-offsets-alist . ((statement-block-intro . +)
     (knr-argdecl-intro . 5)
     (substatement-open . +)
     (label . 0)
     (statement-case-open . +)
     (statement-cont . +)
     (arglist-intro . c-lineup-arglist-intro-after-paren)
     (arglist-close . c-lineup-arglist)
     ))
)

Constructing a custom style is beyond the scope of this book. If you are curious, the long story is available in emacs online info beginning at "Customizing C Indentation." The sample .emacs file given in this chapter (page 241) adds a very simple custom style and arranges to use it on every .c file that is edited.

Comments

Each buffer has its own comment-column variable, which you can view with the CONTROL-H v comment-column RETURN help command. Table 7-19 lists commands that facilitate working with comments.

Table 7-19. Comments

Command

Result

META-;

Inserts a comment on the current line or aligns an existing comment. This command's behavior differs according to the situation.

  • If no comment is on this line, META-; creates an empty comment at the value of comment-column.

  • If text already on this line overlaps the position of comment-column, META-; creates an empty comment one SPACE after the end of the text.

  • If a comment is already on this line but not at the current value of comment-column, META-; realigns the comment at that column. If text is in the way, it places the comment one SPACE after the end of the text.

Once an aligned (possibly empty) comment exists on the line, Point moves to the start of the comment text.

CONTROL-X;

Sets comment-column to the column after Point. The left margin is column 0.

CONTROL-U CONTROL-X;

Kills the comment on the current line. This command sets comment-column from the first comment found above this line and then performs a META-; command to insert or align a comment at that position.

CONTROL-U CONTROL-X;

Sets comment-column to the position of the first comment found above this line and then executes a META-; command to insert or align a comment on this line.

Special-Purpose Modes

The emacs editor includes a third family of Major modes that are not oriented toward a particular language or even toward ordinary editing. Instead, these modes perform some special function. The following modes may define their own key bindings and commands to accomplish that function:

  • Rmail: reads, archives, and composes email

  • Dired: moves around in an ls l display and operates on files

  • VIP: simulates a complete vi environment

  • VC: allows you to drive version-control systems (including RCS, CVS, and Subversion) from within emacs

  • GUD: Grand Unified Debugger; allows you to run and debug C (and other) programs from within emacs

  • Tramp: allows you to edit files on any remote system you can reach with ftp or scp

  • Shell: runs an interactive subshell from inside an emacs buffer

This book discusses only Shell mode.

Shell Mode

One-time shell commands and Region filtering were discussed earlier; refer to "Foreground Shell Commands" on page 227. In Shell mode, however, each emacs buffer has an underlying interactive shell permanently associated with it. This shell takes its input from the last line of the buffer and sends its output back to the buffer, advancing Point as it goes. If you do not edit the buffer, it holds a record of the complete shell session.

The shell runs asynchronously, whether or not you have its buffer in view. The emacs editor uses idle time to read the shell's output and add it to the buffer.

Type META-x shell to create a buffer named *shell* and start a subshell. If a buffer named *shell* already exists, emacs just switches to that buffer. The shell that this command runs is taken from one of the following sources:

  • The Lisp variable explicit-shell-file-name

  • The environment variable ESHELL

  • The environment variable SHELL

To start a second shell, first use META-x rename-buffer to change the name of the existing shell's buffer, and then use META-x shell to start another shell. You can create as many subshells and buffers as you like, all running in parallel.

A special set of commands is defined in Shell mode (Table 7-20). They are bound mostly to two-key sequences starting with CONTROL-C. Each sequence is similar to the ordinary control characters found in Mac OS X but uses a leading CONTROL-C.

Table 7-20. Shell mode

Command

Result

RETURN

If Point is at the end of the buffer, emacs inserts the RETURN and sends this (the last) line to the shell. If Point is elsewhere, it copies this line to the end of the buffer, peeling off the old shell prompt (see the regular expression shell-prompt-pattern), if one existed. Then this copied linenow the last in the bufferis sent to the shell.

CONTROL-C CONTROL-D

Sends CONTROL-D to the shell or its subshell.

CONTROL-C CONTROL-C

Sends CONTROL-C to the shell or its subshell.

CONTROL-C CONTROL\

Sends a quit signal to the shell or its subshell.

CONTROL-C CONTROL-U

Kills the text on the current line not yet completed.

CONTROL-C CONTROL-R

Scrolls back to the beginning of the last shell output, putting the first line of output at the top of the window.

CONTROL-C CONTROL-O

Deletes the last batch of shell output.

Optional: Customizing emacs

At the heart of emacs is a Lisp interpreter written in C. This version of Lisp is significantly extended with many special commands specifically oriented to editing. The interpreter's main task is to execute the Lisp-coded system that implements the look-and-feel of emacs.

Reduced to its essentials, this system implements a continuous loop that watches keystrokes arrive, parses them into commands, executes those commands, and updates the screen. This behavior can be customized in a number of ways.

  • As single keystrokes come in, they are mapped immediately through a keyboard translation table. By changing the entries in this table, it is possible to swap keys. If you are used to vi or vim, you can swap DELETE and CONTROL-H. Then CONTROL-H backspaces as it does in vim, and DELETE (which is not used by vim) is the help key. If you use DELETE as an interrupt key, you may want to choose another key to swap with CONTROL-H.

  • The mapped keystrokes are gathered into small groups called key sequences. A key sequence may be only a single key, such as CONTROL-N, or may include two or more keys, such as CONTROL-X CONTROL-F. Once gathered the key sequences are used to select a particular procedure to be executed. The rules for gathering each key sequence and the specific procedure name to be executed when that sequence comes in are codified in a series of tables called keymaps. By altering the keymaps, you can change the gathering rules or change which procedure is associated with which sequence. If you are used to vi's or vim's use of CONTROL-W to back up over the word you are entering, you may want to change emacs's CONTROL-W binding from the standard kill-region to delete-word-backward.

  • The command behavior is often conditioned by one or more global variables or options. It may be possible to get the behavior you want by setting some of these variables.

  • The command itself is usually a Lisp program that can be reprogrammed to make it behave as desired. Although this task is not appropriate for beginners, the Lisp source to nearly all commands is available and the internal Lisp system is fully documented. As mentioned earlier, it is common practice to load customized Lisp code at startup time, even if you did not write it yourself.

Most emacs documentation glosses over all the translation, gathering, and procedure selection and talks about keystrokes as though they were commands. However, it is still important to know that the underlying machinery exists and to understand that its behavior can be changed.

THE .emacs STARTUP FILE

Each time you start emacs, it loads the file of Lisp code named ~/.emacs. Using this file is the most common way to customize emacs. Two command line options control the use of the .emacs file. The q option ignores the .emacs file so that emacs starts up without it; this is one way to get past a bad .emacs file. The u user option uses the ~user/.emacs file (the .emacs file from the home directory of user).

The .emacs startup file is generally concerned only with key bindings and option settings; it is possible to write the Lisp statements for this file in a straightforward style. Each parenthesized Lisp statement is a Lisp function call. Inside the parentheses the first symbol is the function name; the rest of the SPACE-separated tokens are arguments to that function. The most common function in the .emacs file, setq, is a simple assignment to a global variable. The first argument is the name of the variable to set and the second argument is its value. The following example sets the variable named c-indent-level to 8:

(setq c-indent-level 8)

You can set the default value for a variable that is buffer-private by using the function name setq-default. To set a specific element of a vector, use the function name aset. The first argument is the name of the vector, the second is the target offset, and the third is the value of the target entry. In the startup file the new values are usually constants. Table 7-21 shows the formats of these constants.

Table 7-21. Formats of constants in .emacs

Command

Result

Numbers

Decimal integers, with an optional minus sign

Strings

Similar to C strings but with extensions for CONTROL and META characters: \C-s yields CONTROL-S, \M-s yields META-s, and \M\C-s yields CONTROL-META-s

Characters

Not like C characters; start with ? and continue with a printing character or with a backslash escape sequence (for example, ?a, ?\C-i, ?\033)

Booleans

Not 1 and 0; use t for true and nil for false

Other Lisp objects

Begin with a single quotation mark and continue with the object's name

REMAPPING KEYS

The emacs command loop begins each cycle by translating incoming keystrokes into the name of the command to be executed. The basic translation operation uses the ASCII value of the incoming character to index a 128-element vector called a keymap.

Sometimes a character's eighth bit is interpreted as the META case, but this cannot always be relied on. At the point of translation all META characters appear with the ESCAPE prefix, whether or not they were actually typed that way.

Each position in this vector is one of the following:

  • Not defined at all: No translation possible in this map.

  • The name of another keymap: Switches to that keymap and waits for the next character to arrive.

  • The name of a Lisp function to be called: Translation process is done; call this command.

Because keymaps can reference other keymaps, an arbitrarily complex recognition tree can be set up. The mainstream emacs bindings use at most three keys, with a very small group of well-known prefix keys, each with its well-known keymap name.

Each buffer can have a local keymap that is used first for any keystrokes arriving while a window into that buffer is selected. The local keymap allows the regular mapping to be extended or overridden on a per-buffer basis and is most often used to add bindings for a Major mode.

The basic translation flow runs as follows:

  • Map the first character through the buffer's local keymap. If it is defined as a Lisp function name, translation is done and emacs executes that function. If it is not defined, use this same character to index the global top-level keymap.

  • Map the first character through the top-level global keymap global-map. At this and each following stage, the following conditions hold:

    • If the entry for this character is not defined, it is an error. Send a bell to the terminal and discard all the characters entered in this key sequence.

    • If the entry for this character is defined as a Lisp function name, translation is done and the function is executed.

    • If the entry for this character is defined as the name of another keymap, switch to that keymap and wait for another character to select one of its elements.

Everything must be a command or an error. Ordinary characters that are to be inserted in the buffer are usually bound to the command self-insert-command. Each of the well-known prefix characters is each associated with a keymap (Table 7-22).

Table 7-22. Keymap prefixes

Keymap prefix

Applies to

ctl-x-map

Characters following CONTROL-X

ctl-x-4-map

Characters following CONTROL-X 4

esc-map

Characters following ESCAPE (including META characters)

help-map

Characters following CONTROL-H

mode-specific-map

Characters following CONTROL-C

To see the current state of the keymaps, type CONTROL-H b. They appear in the following order: local, global, and shorter maps for each prefix key. Each line specifies the name of the Lisp function to be called; the documentation for that function can be retrieved with the commands CONTROL-H f function-name or CONTROL-H k key-sequence.

The most common type of keymap customization is making small changes to the global command assignments without creating any new keymaps or commands. This type of customization is most easily done in the .emacs file using the Lisp function define-key. The define-key function takes three arguments:

  • The keymap name

  • A single character defining a position in that map

  • The command to be executed when this character appears

For instance, to bind the command backward-kill-word to CONTROL-W, use the statement

(define-key global-map "\C-w" 'backward-kill-word)

To bind the command kill-region to CONTROL-X CONTROL-K, use the statement

(define-key ctl-x-map "\C-k" 'kill-region)

The \ character causes C-w to be interpreted as CONTROL-W instead of three letters (equivalent to \^w). The unmatched single quotation mark in front of the command name is correct. This Lisp escape character keeps the name from being evaluated too soon.

A SAMPLE .emacs FILE

The following ~/.emacs file produces a plain editing environment that minimizes surprises for vi and vim users. Of course, if any section or any line is inapplicable or not to your liking, you can edit it out or comment it with one or more ; comment characters, beginning in column 1.

[View full width]
;;; Preference Variables

(setq make-backup-files nil)          ;Do not make backup files
(setq backup-by-copying t)            ;If you do, at least do not
   destroy links
(setq delete-auto-save-files t)       ;Delete autosave files when
   writing orig
(setq blink-matching-paren nil)       ;Do not blink opening delim
(setq-default case-fold-search nil)   ;Do not fold cases in search
(setq require-final-newline 'ask)     ;Ask about missing final
   newline

;; Reverse mappings for C-h and DEL.
(keyboard-translate ?\C-h ?\177)
(keyboard-translate ?\177 ?\C-h)

;; reassigning C-w to keep on deleting words backward

;; C-w is supposed to be kill-region, but it's a great burden for
   vi-trained fingers.
;; Bind it instead to backward-kill-word for more familiar,
   friendly behavior.
(define-key global-map "\^w" 'backward-kill-word)

;; for kill-region use a two-key sequence c-x c-k.
(define-key ctl-x-map "\^k" 'kill-region)

;; C mode customization: set vanilla (8-space bsd) indention style

(require 'cc-mode)                          ;kiss: be sure it's here

(c-add-style                                ;add indentation style
 "bsd8"                                     ;old bsd (8 spaces)
 '((c-basic-offset . 8)
   (c-hanging-comment-ender-p . nil)    ;isolated "*/" ends blk
   comments
   (c-comment-only-line-offset . 0)
   (c-offsets-alist . ((statement-block-intro . +)
                       (knr-argdecl-intro . +)
                       (substatement-open . 0)
                       (label . 0)
                       (statement-cont . +)
                       ))
   ))
(add-hook                                ;this is our default style,
 'c-mode-hook                            ;set it always in
   c-mode-hook
 (function
  (lambda ()
    (c-set-style "bsd8"))))

;; end of c mode style setup


More Information

A lot of emacs documentation is available in both paper and electronic form. The GNU emacs Web page is a good place to start: www.gnu.org/software/emacs.

The comp.emacs and gnu.emacs.help newsgroups offer support for and a general discussion about emacs.

Access to emacs

The emacs editor is included with Mac OS X. You can download the latest version of the source code from www.gnu.org.

The Free Software Foundation can be reached at these addresses:

Mail: | Free Software Foundation, Inc.
 | 59 Temple Place, Suite 330
 | Boston, MA 02111-1307, USA
E-mail: | gnu@gnu.org
Phone: | 1 617-542-5942


Chapter Summary

You can precede many of the commands in the following tables with a numeric argument to make the command repeat the number of times specified by the argument. Precede a numeric argument with CONTROL-U to keep emacs from entering the argument as text.

Table 7-23 lists commands that move the cursor.

Table 7-23. Moving the cursor

Command

Result

CONTROL-F

Forward by characters

CONTROL-B

Backward by characters

META-f

Forward by words

META-b

Backward by words

META-e

To end of sentence

META-a

To beginning of sentence

META-}

To end of paragraph

META-{

To beginning of paragraph

META->

Forward to end of buffer

META-<

Backward to beginning of buffer

CONTROL-ESCAPE

To end of line

CONTROL-A

To beginning of line

CONTROL-N

Forward (down) one line

CONTROL-P

Backward (up) one line

CONTROL-V

Scroll forward (down) one window

META-v

Scroll backward (up) one window

CONTROL-L

Clear and repaint screen, and scroll current line to center of window

META-r

To beginning of middle line

CONTROL-U num META-r

To beginning of line number num (0 = top, = bottom)

Table 7-24 lists commands that kill and delete text.

Table 7-24. Killing and deleting text

Command

Result

CONTROL-DELETE

Deletes character under cursor

DELETE

Deletes character to left of cursor

META-d

Kills forward to end of current word

META-DELETE

Kills backward to beginning of previous word

META-k

Kills forward to end of sentence

CONTROL-X DELETE

Kills backward to beginning of sentence

CONTROL-K

Kills forward to, but not including, line-ending LINEFEED; if there is no text between the cursor and the LINEFEED, kills the LINEFEED

CONTROL-U 1 CONTROL-K

Kills from cursor forward to and including LINEFEED

CONTROL-U 0 CONTROL-K

Kills from cursor backward to beginning of line

META-z char

Kills forward to, but not including, next occurrence of char

META-w

Copies Region to Kill Ring (does not delete Region from buffer)

CONTROL-W

Kills Region (deletes Region from buffer)

CONTROL-Y

Yanks most recently killed text into current buffer at Point; sets Mark at beginning of this text, with Point and cursor at the end

META-y

Erases just-yanked text, rotates Kill Ring, and yanks next item (only after CONTROL-Y or META-y)

Table 7-25 lists commands that search for strings and regular expressions.

Table 7-25. Search commands

Command

Result

CONTROL-S

Prompts incrementally for a string and searches forward

CONTROL-S RETURN

Prompts for a complete string and searches forward

CONTROL-R

Prompts incrementally for a string and searches backward

CONTROL-R RETURN

Prompts for a complete string and searches backward

META-CONTROL-S

Prompts incrementally for a regular expression and searches forward

META-CONTROL-S RETURN

Prompts for a complete regular expression and searches forward

META-x isearch-backward-regexp

Prompts incrementally for a regular expression and searches forward

META-x isearch-backward-regexp RETURN

Prompts for a complete regular expression and searches backward

Table 7-26 lists commands that provide online help.

Table 7-26. Online help

Command

Result

CONTROL-H a

Prompts for string and displays a list of commands whose names contain string

CONTROL-H b

Displays a (long) table of all key bindings now in effect

CONTROL-H c key-sequence

Displays the name of the command bound to key-sequence

CONTROL-H k key-sequence

Displays the name of and documentation for the command bound to key-sequence

CONTROL-H f

Prompts for the name of a Lisp function and displays the documentation for that function

CONTROL-H i (lowercase "i")

Displays the top menu of info (page 30)

CONTROL-H l (lowercase "l")

Displays the last 100 characters typed

CONTROL-H m

Displays the documentation and special key bindings for the current Major mode

CONTROL-H n

Displays the emacs news file

CONTROL-H t

Starts an emacs tutorial session

CONTROL-H v

Prompts for a Lisp variable name and displays the documentation for that variable

CONTROL-H w

Prompts for a command name and displays the key sequence, if any, bound to that command

Table 7-27 lists commands that work with a Region.

Table 7-27. Working with a Region

Command

Result

META-W

Copies Region nondestructively to the Kill Ring

CONTROL-W

Kills (deletes) Region

META-x print-region

Copies Region to the print spooler

META-x append-to-buffer

Prompts for buffer name and appends Region to that buffer

META-x append-to-file

Prompts for filename and appends Region to that file

CONTROL-X CONTROL-U

Converts Region to uppercase

CONTROL-X CONTROL-L

Converts Region to lowercase

Table 7-28 lists commands that work with lines.

Table 7-28. Working with lines

Command

Result

META-x occur

Prompts for a regular expression and lists each line containing a match for the expression in a buffer named *Occur*

META-x delete-matching-lines

Prompts for a regular expression and deletes lines from Point forward that have a match for the regular expression

META-x delete-non-matching-lines

Prompts for a regular expression and deletes lines from Point forward that do not have a match for the regular expression

Table 7-29 lists commands that replace strings and regular expressions unconditionally and interactively.

Table 7-29. Commands that replace text

Command

Result

META-x replace-string

Prompts for two strings and replaces each instance of the first string with the second string from Mark forward; sets Mark at the start of the command

META-% or META-x query-replace

As above but queries for each replacement (see Table 7-30 for a list of responses)

META-x replace-regexp

Prompts for a regular expression and a string, and replaces each match for the regular expression with the string; sets Mark at the start of the command

META-x query-replace-regexp

As above but queries for each replacement (see Table 7-30 for a list of responses)

Table 7-30 lists responses to replacement queries.

Table 7-30. Responses to replacement queries

Command

Result

RETURN

Quits searching (does not make or query for any more replacements)

SPACE

Makes this replacement and continues querying

DELETE

Does not make this replacement and continues querying

, (comma)

Makes this replacement, displays the result, and asks for another command

. (period)

Makes this replacement and does not make or query for any more replacements

! (exclamation point)

Replaces this and all remaining instances without querying

Table 7-31 lists commands that work with windows.

Table 7-31. Working with windows

Command

Result

CONTROL-X b

Prompts for and displays a different buffer in current window

CONTROL-X 2

Splits current window vertically into two

CONTROL-X 3

Splits current window horizontally into two

CONTROL-X 0 (lowercase "o")

Selects other window

META-CONTROL-V

Scrolls other window

CONTROL-X 4b

Prompts for buffer name and selects it in other window

CONTROL-X 4f

Prompts for filename and selects it in other window

CONTROL-X 0 (zero)

Deletes current window

CONTROL-X 1 (one)

Deletes all windows except current window

META-x shrink-window

Makes current window one line shorter

CONTROL-X ^

Makes current window one line taller

CONTROL-X }

Makes current window one character wider

CONTROL-X {

Makes current window one character narrower

Table 7-32 lists commands that work with files.

Table 7-32. Working with files

Command

Result

CONTROL-X CONTROL-F

Prompts for a filename and reads its contents into a new buffer; assigns the file's simple filename as the buffer name.

CONTROL-X CONTROL-V

Prompts for a filename and reads its contents into the current buffer (overwriting the contents of the current buffer).

CONTROL-X 4 CONTROL-F

Prompts for a filename and reads its contents into a new buffer; assigns the file's simple filename as the buffer name. Creates a new window for the new buffer and selects that window. This command splits the screen in half if you begin with only one window.

CONTROL-X CONTROL-S

Saves the current buffer to the original file.

CONTROL-X S

Prompts for whether to save each modified buffer (y/n).

META-x set-visited-file-name

Prompts for a filename and sets the current buffer's "original" name to that filename.

CONTROL-X CONTROL-W

Prompts for a filename, sets the current buffer's "original" name to that filename, and saves the current buffer in that file.

META-~ (tilde)

Clears modified flag from the current buffer. Use with caution.

Table 7-33 lists commands that work with buffers.

Table 7-33. Working with buffers

Command

Result

CONTROL-X CONTROL-S

Saves current buffer in its associated file.

CONTROL-X CONTROL-F

Prompts for filename and visits (opens) that file.

CONTROL-X b

Prompts for buffer name and selects it. If that buffer does not exist, creates it.

CONTROL-X 4b

Prompts for buffer name and displays that buffer in another window. The existing window is not disturbed, although the new window may overlap it.

CONTROL-X CONTROL-B

Creates a buffer named *Buffer List* and displays it in another window. The existing window is not disturbed, although the new window may overlap it. The new buffer is not selected. In the *Buffer List* buffer, each buffer's data is displayed with its name, size, mode(s), and original filename.

META-x rename-buffer

Prompts for a new buffer name and assigns this new name to the current buffer.

CONTROL-X CONTROL-Q

Toggles the current buffer's readonly status and the associated %% Mode Line indicator.

META-x append-to-buffer

Prompts for buffer name and appends Region to the end of that buffer.

META-x prepend-to-buffer

Prompts for buffer name and prepends Region to beginning of that buffer.

META-x copy-to-buffer

Prompts for buffer name, deletes contents of that buffer, and copies the Region to that buffer.

META-x insert-buffer

Prompts for buffer name and inserts entire contents of that buffer in current buffer at Point.

CONTROL-X k

Prompts for buffer name and deletes that buffer.

META-x kill-some-buffers

Goes through the entire buffer list and offers the chance to delete each buffer.

Table 7-34 lists commands that run shell commands in the foreground. These commands may not work with all shells.

Table 7-34. Foreground shell commands

Command

Result

META-! (exclamation point)

Prompts for shell command, executes it, and displays the output

CONTROL-U META-! (exclamation point)

Prompts for shell command, executes it, and inserts the output at Point

META-| (vertical bar)

Prompts for shell command, supplies Region as input to that command, and displays output of command

CONTROL-U META-| (vertical bar)

Prompts for shell command, supplies Region as input to that command, deletes old Region, and inserts output of command in place of Region

Table 7-35 lists commands that run shell commands in the background.

Table 7-35. Background shell commands

Command

Result

META-x compile

Prompts for shell command and then runs that command in the background, with output going to the buffer named *compilation*

META-x kill-compilation

Kills background process

Table 7-36 lists commands that convert text from uppercase to lowercase and vice versa.

Table 7-36. Case conversion commands

Command

Result

META-l (lowercase "l")

Converts word to right of Point to lowercase

META-u

Converts word to right of Point to uppercase

META-c

Converts word to right of Point to initial caps

CONTROL-X CONTROL-L

Converts Region to lowercase

CONTROL-X CONTROL-U

Converts Region to uppercase

Table 7-37 lists commands that work in C mode.

Table 7-37. C mode commands

Command

Result

CONTROL-META-f

Moves forward over expression

CONTROL-META-b

Moves backward over expression

CONTROL-META-k

Moves forward over expression and kills it

CONTROL-META-@

Sets Mark at the position CONTROL-META-f would move to, without changing Point

CONTROL-META-a

Moves to beginning of the most recent function definition

CONTROL-META-e

Moves to the end of the next function definition

CONTROL-META-h

Moves Point to beginning and Mark to end of current (or next, if between) function definition

Type META-x shell to create a buffer named *shell* and start a subshell. Table 7-38 lists commands that work on this buffer.

Table 7-38. Shell mode commands

Command

Result

RETURN

Sends current line to the shell

CONTROL-C CONTROL-D

Sends CONTROL-D to shell or its subshell

CONTROL-C CONTROL-C

Sends CONTROL-C to shell or its subshell

CONTROL-C CONTROL-\

Sends quit signal to shell or its subshell

CONTROL-C CONTROL-U

Kills text on the current line not yet completed

CONTROL-C CONTROL-R

Scrolls back to beginning of last shell output, putting first line of output at the top of the window

CONTROL-C CONTROL-O (uppercase "O")

Deletes last batch of shell output


Exercises

1.

Given a buffer full of English text, answer the following questions:

  1. How would you change every instance of his to hers?

  2. How would you do this only in the final paragraph?

  3. Is there a way to look at every usage in context before changing it?

  4. How would you deal with the possibility that His might begin a sentence?

2.

Which command moves the cursor to the end of the current paragraph? Can you use this command to skip through the buffer in one-paragraph steps?

3.

Suppose that you get lost in the middle of typing a long sentence.

  1. Is there an easy way to kill the botched sentence and start over?

  2. What if only one word is incorrect? Is there an alternative to backspacing one letter at a time?

4.

After you have been working on a paragraph for a while, most likely some lines will have become too short and others too long. Is there a command to "neaten up" the paragraph without rebreaking all the lines by hand?

5.

Is there a way to change the entire contents of the buffer to capital letters? Can you think of a way to change just one paragraph?

6.

How would you reverse the order of two paragraphs?

7.

How would you reverse two words?

8.

Imagine that you saw a Usenet posting with something particularly funny in it and saved the posting to a file. How would you incorporate this file into your own buffer? What if you wanted only a couple of paragraphs? How would you add > to the beginning of each included line?

9.

On the keyboard alone emacs has always offered a full set of editing possibilities. Generally several techniques will accomplish the same goal for any editing task. In the X environment the choice is enlarged still further with a new group of mouse-oriented visual alternatives. From these options you must select the way that you like to solve a given editing puzzle best.

Consider this Shakespearean fragment:

1. Full fathom five thy father lies;
2.   Of his bones are coral made;
3. Those are pearls that were his eyes:
4.   Nothing of him that doth fade,
5. But doth suffer a sea-change
6. Into something rich and strange.
7. Sea-nymphs hourly ring his knell:
8.           Ding-dong.
9. Hark! now I hear them--
10.      Ding-dong, bell!

The following fragment has been typed with some errors:

1. Full fathiom five tyy father lies;
2. These are pearls that were his eyes:
3.   Of his bones are coral made;
4.   Nothin of him that doth fade,
5. But doth susffer a sea-change
6. Into something rich and strange.
7. Sea-nymphs hourly ring his knell:
8.           Ding=dong.
9. Hard! now I hear them--
10.      Ding-dong, bell!

Use only the keyboard to answer the following:

  1. How many ways can you think of to move the cursor to the spelling errors?

  2. Once the cursor is on or near the errors, how many ways can you think of to fix them?

  3. Lines 2 and 3 are transposed. How many ways can you think of to correct this situation?


Advanced Exercises

10.

Assume that your buffer contains the C code shown here, with the Major mode set for C and the cursor positioned at the end of the while line as shown by the black square:

/*
* Copy string s2 to s1.  s1 must be large enough
* return s1
*/
char *
strcpy(s1, s2)
register char *s1, *s2;
{
     register char *os1;

     os1 = s1;
     while (*s1++ = *s2++)
     ;
return(os1);
}

/* Copy source into dest, stopping after '\0' is copied, and
   return a pointer to the '\0' at the end of dest.  Then our caller
   can concatenate to the dest string without another strlen call. */
char *
stpcpy (dest, source)
     char *dest;
     char *source;
{
 while ((*dest++ = *source++) != '\0')
    ; /* void loop body */
 return (dest - 1);
}

  1. Which command moves the cursor to the opening brace of strcpy? Which command moves the cursor past the closing brace? Can you use these commands to skip through the buffer in one-procedure steps?

  2. Assume the cursor is just past the closing parenthesis of the while condition. How do you move to the matching opening parenthesis? How do you move back to the matching close parenthesis again? Does the same command set work for matched [] and {}? How does this differ from the vim % command?

  3. One procedure is indented in the Berkeley indention style; the other is indented in the GNU style. Which command reindents a line in accordance with the current indention style you have set up? How would you reindent an entire procedure?

  4. Suppose that you want to write five string procedures and intend to use strcpy as a starting point for further editing. How would you make five copies of the strcpy procedure?

  5. How would you compile the code without leaving emacs?


Part III: The Shells

CHAPTER 8 THE BOURNE AGAIN SHELL

CHAPTER 9 THE TC SHELL


Chapter 8. The Bourne Again Shell

Previous Page
Next Page

8. The Bourne Again Shell

IN THIS CHAPTER

Startup Files | 259
Redirecting Standard Error | 262
Writing a Simple Shell Script | 265
Job Control | 272
Manipulating the Directory Stack | 275
Parameters and Variables | 278
Processes | 293
History | 295
Reexecuting and Editing Commands | 296
Aliases | 311
Functions | 314
Controlling bash Features and Options | 316
Processing the Command Line | 321

This chapter picks up where Chapter 5 left off by focusing on the Bourne Again Shell (bash). It notes where the TC Shell (tcsh) implementation of a feature differs from that of bash; if appropriate, you are directed to the page where the alternative implementation is discussed. Chapter 13 expands on this chapter, exploring control flow commands and more advanced aspects of programming the Bourne Again Shell. The bash home page is www.gnu.org/software/bash. The bash info page is a complete Bourne Again Shell reference.

The Bourne Again Shell and TC Shell are command interpreters and high-level programming languages. As command interpreters, they process commands you enter on the command line in response to a prompt. When you use the shell as a programming language, it processes commands stored in files called shell scripts. Like other languages, shells have variables and control flow commands (for example, for loops and if statements).

When you use a shell as a command interpreter, you can customize the environment you work in. You can make your prompt display the name of the working directory, create a function or alias for cp that keeps it from overwriting certain kinds of files, take advantage of keyword variables to change aspects of how the shell works, and so on. You can also write shell scripts that do your bidding, from a one-line script that stores a long, complex command to a longer script that runs a set of reports, prints them, and mails you a reminder when the job is done. More complex shell scripts are themselves programs; they do not just run other programs. Chapter 13 has some examples of these types of scripts.

Most system shell scripts are written to run under the Bourne Again Shell. If you will ever work in single-user modeas when you boot your system or do system maintenance, administration, or repair work, for exampleit is a good idea to become familiar with this shell.

This chapter expands on the interactive features of the shell described in Chapter 5, explains how to create and run simple shell scripts, discusses job control, introduces the basic aspects of shell programming, talks about history and aliases, and describes command line expansion. Chapter 9 covers interactive use of the TC Shell and TC Shell programming, and Chapter 13 presents some more challenging shell programming problems.


Background

The Bourne Again Shell is based on the Bourne Shell (the early UNIX shell; this book refers to it as the original Bourne Shell to avoid confusion), which was written by Steve Bourne of AT&T's Bell Laboratories. Over the years the original Bourne Shell has been expanded but it remains the basic shell provided with many commercial versions of UNIX.

sh Shell

Because of its long and successful history, the original Bourne Shell has been used to write many of the shell scripts that help manage UNIX systems. Some of these scripts appear in Mac OS X as Bourne Again Shell scripts. Although the Bourne Again Shell includes many extensions and features not found in the original Bourne Shell, bash maintains compatibility with the original Bourne Shell so you can run Bourne Shell scripts under bash. On UNIX systems the original Bourne Shell is named sh. On Mac OS X sh is a copy of bash ensuring that scripts that require the presence of the Bourne Shell still run. When called as sh, bash does its best to emulate the original Bourne Shell.

Korn Shell

System V UNIX introduced the Korn Shell (ksh), written by David Korn. This shell extended many features of the original Bourne Shell and added many new features. Some features of the Bourne Again Shell, such as command aliases and command line editing, are based on similar features from the Korn Shell. Recent releases of Mac OS X include the Korn Shell.

POSIX standards

The POSIX (the Portable Operating System Interface) family of related standards is being developed by PASC (IEEE's Portable Application Standards Committee, www.pasc.org). A comprehensive FAQ on POSIX, including many links, appears at www.opengroup.org/austin/papers/posix_faq.html.

POSIX standard 1003.2 describes shell functionality. The Bourne Again Shell provides the features that match the requirements of this POSIX standard. Efforts are under way to make the Bourne Again Shell fully comply with the POSIX standard. In the meantime, if you invoke bash with the --posix option, the behavior of the Bourne Again Shell will more closely match the POSIX requirements.


Shell Basics

This section covers writing and using startup files, redirecting standard error, writing and executing simple shell scripts, separating and grouping commands, implementing job control, and manipulating the directory stack.

Startup Files

When a shell starts, it runs startup files to initialize itself. Which files the shell runs depends on whether it is a login shell, an interactive shell that is not a login shell (such as you get by giving the command bash), or a noninteractive shell (one used to execute a shell script). You must have read access to a startup file to execute the commands in it. Mac OS X puts appropriate commands in some of these files. This section covers bash startup files. See page 340 for information on tcsh startup files.

Login Shells

The files covered in this section are executed by login shells and shells that you start with the login option. Login shells are, by their nature, interactive.

/etc/profile

The shell first executes the commands in /etc/profile. By default when called from bash, a command in this file calls /etc/bashrc. Superuser can set up this file to establish systemwide default characteristics for bash users.

.bash_profile .bash_login .profile

Next the shell looks for ~/.bash_profile, ~/.bash_login, and ~/.profile (~/ is shorthand for your home directory), in that order, executing the commands in the first of these files it finds. You can put commands in one of these files to override the defaults set in /etc/profile.

.bash_logout

When you log out, bash executes commands in the ~/.bash_logout file. Frequently commands that clean up after a session, such as those that remove temporary files, go in this file.

Interactive Nonlogin Shells

The commands in the preceding startup files are not executed by interactive, nonlogin shells. However, these shells inherit from the login shell variables that are set by these startup files.

/etc/bashrc

Although not called by bash directly, many ~/.bashrc files call /etc/bashrc. This setup allows Superuser to establish systemwide default characteristics for nonlogin bash shells.

.bashrc

An interactive nonlogin shell executes commands in the ~/.bashrc file. Typically a startup file for a login shell, such as .bash_profile, runs this file, so that both login and nonlogin shells benefit from the commands in .bashrc.

Noninteractive Shells

The commands in the previously described startup files are not executed by noninteractive shells, such as those that runs shell scripts. However, these shells inherit from the login shell variables that are set by these startup files.

BASH_ENV

Noninteractive shells look for the environment variable BASH_ENV (or ENV, if the shell is called as sh) and execute commands in the file named by this variable.

Setting Up Startup Files

Although many startup files and types of shells exist, usually all you need are the .bash_profile and .bashrc files in your home directory. Commands similar to the following in .bash_profile run commands from .bashrc for login shells (when .bashrc exists). With this setup, the commands in .bashrc are executed by login and nonlogin shells.

if [ -f ~/.bashrc ]; then source ~/.bashrc; fi

The [ -f ~/.bashrc ] tests whether the file named .bashrc in your home directory exists. See page 871 for more information on test and its synonym [].

Tip: Use .bash_profile to set PATH

Because commands in .bashrc may be executed many times, and because subshells inherit exported variables, it is a good idea to put commands that add to existing variables in the .bash_profile file. For example, the following command adds the bin subdirectory of the home directory to PATH (page 285) and should go in .bash_profile:

PATH=$PATH:$HOME/bin

When you put this command in .bash_profile and not in .bashrc, the string is added to the PATH variable only once, when you log in.

Modifying a variable in .bash_profile allows changes you make in an interactive session to propagate to subshells. In contrast, modifying a variable in .bashrc overrides changes inherited from a parent shell.

Sample .bash_profile and .bashrc files follow. Some of the commands used in these files are not covered until later in this chapter. In any startup file, you must export variables and functions that you want to be available to child processes. For more information refer to "Locality of Variables" on page 560.

$ cat ~/.bash_profile
if [ -f ~/.bashrc ]; then
    source ~/.bashrc          # read local startup file if it exists
fi
PATH=$PATH:.                  # add the working directory to PATH
export PS1='[\h \W \!]\$ '    # set prompt

The first command in the preceding .bash_profile file executes the commands in the user's .bashrc file if it exists. The next command adds to the PATH variable (page 285). Typically PATH is set and exported in /etc/profile so it does not need to be exported in a user's startup file. The final command sets and exports PS1 (page 286), which controls the user's prompt.

Next is a sample .bashrc file. The first command executes the commands in the /etc/bashrc file if it exists. Next the VIMINIT (page 180) variable is set and exported and several aliases (page 311) are established. The final command defines a function (page 314) that swaps the names of two files.

$ cat ~/.bashrc
if [ -f /etc/bashrc ]; then
    source /etc/bashrc             # read global startup file if it exists
fi
set -o noclobber                   # prevent overwriting files
unset MAILCHECK                    # turn off "you have new mail" notice
export VIMINIT='set ai aw'         # set vim options
alias df='df -h'                   # set up aliases
alias rm='rm -i'                   # always do interactive rm's
alias lt='ls -ltrh | tail'
alias h='history | tail'
alias ch='chmod 755 '

function switch()                  # a function to exchange the names
{                                  # of two files
   local tmp=$$switch
   mv "$1" $tmp
   mv "$2" "$1"
   mv $tmp "$2"
}

. (Dot) or source: Runs a Startup File in the Current Shell

After you edit a startup file such as .bashrc, you do not have to log out and log in again to put the changes into effect. You can run the startup file using the . (dot) or source builtin (they are the same command under bash; only source is available under tcsh [page 376], and only the dot builtin is available under ksh). As with all other commands, the . must be followed by a SPACE on the command line. Using the . or source builtin is similar to running a shell script, except that these commands run the script as part of the current process. Consequently, when you use . or source to run a script, changes you make to variables from within the script affect the shell that you run the script from. You can use the . or source command to run any shell scriptnot just a startup filebut undesirable side effects (such as changes in the values of shell variables you rely on) may occur. If you ran a startup file as a regular shell script and did not use the . or source builtin, the variables created in the startup file would remain in effect only in the subshell running the scriptnot in the shell you ran the script from. For more information refer to "Locality of Variables" on page 560.

In the following example, .bashrc sets several variables and sets PS1, the prompt, to the name of the host. The . builtin puts the new values into effect.

$ cat ~/.bashrc
export TERM=vt100             # set the terminal type
export PS1="$(hostname -f): " # set the prompt string
export CDPATH=:$HOME          # add HOME to CDPATH string
stty kill '^u'                # set kill line to control-u
$ . ~/.bashrc
bravo.example.com:

Commands That Are Symbols

The Bourne Again Shell uses the symbols (, ), [, ], and $ in a variety of ways. To minimize confusion, Table 8-1 lists the most common use of each of these symbols, even though some of them are not introduced until later.

Table 8-1. Builtin commands that are symbols

Symbol

Command

( )

Subshell (page 271)

$( )

Command substitution (page 327)

(( ))

Arithmetic evaluation; a synonym for let (use when the enclosed value contains an equal sign) (page 585)

$(( ))

Arithmetic expansion (not for use with an enclosed equal sign) (page 325)

[ ]

The test command. (pages 525, 527, 540, and 871)

[[ ]]

Conditional expression; similar to [ ] but adds string comparisons (page 586)

Redirecting Standard Error

Chapter 5 covered the concept of standard output and explained how to redirect standard output of a command. In addition to standard output, commands can send output to standard error. A command can send error messages to standard error to keep them from getting mixed up with the information it sends to standard output.

Just as it does with standard output, by default the shell sends a command's standard error to the screen. Unless you redirect one or the other, you may not know the difference between the output a command sends to standard output and the output it sends to standard error. This section covers the syntax used by the Bourne Again Shell. See page 346 if you are using the TC Shell.

File descriptors

A file descriptor is the place a program sends its output to and gets its input from. When you execute a program, the process running the program opens three file descriptors: 0 (standard input), 1 (standard output), and 2 (standard error). The redirect output symbol (> [page 122]) is shorthand for 1>, which tells the shell to redirect standard output. Similarly < (page 124) is short for 0<, which redirects standard input. The symbols 2> redirect standard error. For more information refer to "File Descriptors" on page 555.

The following examples demonstrate how to redirect standard output and standard error to different files and to the same file. When you run the cat utility with the name of a file that does not exist and the name of a file that does exist, cat sends an error message to standard error and copies the file that does exist to standard output. Unless you redirect them, both messages appear on the screen.

$ cat y
This is y.
$ cat x
cat: x: No such file or directory

$ cat x y
cat: x: No such file or directory
This is y.

When you redirect standard output of a command, output sent to standard error is not affected and still appears on the screen.

$ cat x y > hold
cat: x: No such file or directory
$ cat hold
This is y.

Similarly, when you send standard output through a pipe, standard error is not affected. The following example sends standard output of cat tHRough a pipe to tr (page 879), which in this example converts lowercase characters to uppercase. The text that cat sends to standard error is not translated because it goes directly to the screen rather than through the pipe.

$ cat x y | tr "[a-z]" "[A-Z]"
cat: x: No such file or directory
THIS IS Y.

The following example redirects standard output and standard error to different files. The notation 2> tells the shell where to redirect standard error (file descriptor 2). The 1> tells the shell where to redirect standard output (file descriptor 1). You can use > in place of 1>.

$ cat x y 1> hold1 2> hold2
$ cat hold1
This is y.
$ cat hold2
cat: x: No such file or directory

Duplicating a file descriptor

In the next example, 1> redirects standard output to hold. Then 2>&1 declares file descriptor 2 to be a duplicate of file descriptor 1. As a result both standard output and standard error are redirected to hold.

$ cat x y 1> hold 2>&1
$ cat hold
cat: x: No such file or directory
This is y.

In the preceding example, 1> hold precedes 2>&1. If they had been listed in the opposite order, standard error would have been made a duplicate of standard output before standard output was redirected to hold. In that case only standard output would have been redirected to hold.

The next example declares file descriptor 2 to be a duplicate of file descriptor 1 and sends the output for file descriptor 1 through a pipe to the tr command.

$ cat x y 2>&1 | tr "[a-z]" "[A-Z]"
CAT: X: NO SUCH FILE OR DIRECTORY
THIS IS Y.

Sending errors to standard error

You can also use 1>&2 to redirect standard output of a command to standard error. This technique is often used in shell scripts to send the output of echo to standard error. In the following script, standard output of the first echo is redirected to standard error:

$ cat message_demo
echo This is an error message. 1>&2
echo This is not an error message.

If you redirect standard output of message_demo, error messages such as the one produced by the first echo will still go to the screen because you have not redirected standard error. Because standard output of a shell script is frequently redirected to another file, you can use this technique to display on the screen error messages generated by the script. The lnks script (page 532) uses this technique. You can also use the exec builtin to create additional file descriptors and to redirect standard input, standard output, and standard error of a shell script from within the script (page 574).

The Bourne Again Shell supports the redirection operators shown in Table 8-2.

Table 8-2. Redirection operators

Operator

Meaning

< filename

Redirects standard input from filename.

> filename

Redirects standard output to filename unless filename exists and noclobber (page 125) is set. If noclobber is not set, this redirection creates filename if it does not exist.

>| filename

Redirects standard output to filename, even if the file exists and noclobber (page 125) is set.

>> filename

Redirects and appends standard output to filename unless filename exists and noclobber (page 125) is set. If noclobber is not set, this redirection creates filename if it does not exist.

<&m

Duplicates standard input from file descriptor m (page 555).

[n]>&m

Duplicates standard output or file descriptor n if specified from file descriptor m (page 555).

[n]<&-

Closes standard input or file descriptor n if specified (page 555).

[n]>&-

Closes standard output or file descriptor n if specified.

Writing a Simple Shell Script

A shell script is a file that contains commands that the shell can execute. The commands in a shell script can be any commands you can enter in response to a shell prompt. For example, a command in a shell script might run a Mac OS X utility, a compiled program, or another shell script. Like the commands you give on the command line, a command in a shell script can use ambiguous file references and can have its input or output redirected from or to a file or sent through a pipe (page 128). You can also use pipes and redirection with the input and output of the script itself.

In addition to the commands you would ordinarily use on the command line, control flow commands (also called control structures) find most of their use in shell scripts. This group of commands enables you to alter the order of execution of commands in a script just as you would alter the order of execution of statements using a structured programming language. Refer to "Control Structures" on page 524 (bash) and page 364 (tcsh) for specifics.

The shell interprets and executes the commands in a shell script, one after another. Thus a shell script enables you to simply and quickly initiate a complex series of tasks or a repetitive procedure.

chmod: Makes a File Executable

To execute a shell script by giving its name as a command, you must have permission to read and execute the file that contains the script (refer to "Access Permissions" on page 87). Read permission enables you to read the file that holds the script. Execute permission tells the shell and the system that the owner, group, and/or public has permission to execute the file; it implies that the content of the file is executable.

When you create a shell script using an editor, the file does not typically have its execute permission set. The following example shows a file named whoson that contains a shell script:

$ cat whoson
date
echo "Users Currently Logged In"
who

$ whoson
bash: ./whoson: Permission denied

You cannot execute whoson by giving its name as a command because you do not have execute permission for the file. The shell does not recognize whoson as an executable file and issues an error message when you try to execute it. When you give the filename as an argument to bash (bash whoson), bash takes the argument to be a shell script and executes it. In this case bash is executable and whoson is an argument that bash executes so you do not need to have permission to execute whoson. You can do the same with tcsh script files.

Tip: Command not found?

If you get the message

$ whoson
bash: whoson: command not found

the shell is not set up to search for executable files in the working directory. Give this command instead:

$ ./whoson

The ./tells the shell explicitly to look for an executable file in the working directory. To change the environment so that the shell searches the working directory automatically, see page 285.

The chmod utility changes the access privileges associated with a file. Figure 8-1 shows ls with the -l option displaying the access privileges of whoson before and after chmod gives execute permission to the file's owner.

Figure 8-1. Using chmod to make a shell script executable

The first ls displays a hyphen (-) as the fourth character, indicating that the owner does not have permission to execute the file. Next chmod gives the owner execute permission: The u+x causes chmod to add (+) execute permission (x) for the owner (u). (The u stands for user, although it means the owner of the file who may be the user of the file at any given time.) The second argument is the name of the file. The second ls shows an x in the fourth position, indicating that the owner now has execute permission.

If other users will execute the file, you must also change group and/or public access permissions for the file. Any user must have execute access to use the file's name as a command. If the file is a shell script, the user trying to execute the file must also have read access to the file. You do not need read access to execute a binary executable (compiled program).

The final command in Figure 8-1 shows the shell executing the file when its name is given as a command. For more information refer to "Access Permissions" on page 87 and to ls and chmod in Part V.

#! Specifies a Shell

You can put a special sequence of characters on the first line of a file to tell the operating system which shell should execute the file. Because the operating system checks the initial characters of a program before attempting to exec it, these characters save the system from making an unsuccessful attempt. If #! are the first two characters of a script, the system interprets the characters that follow as the absolute pathname of the utility that should execute the script. This can be the pathname of any program, not just a shell. The following example specifies that bash should run the script:

$ cat bash_script
#!/bin/bash
echo "This is a Bourne Again Shell script."

The #! characters are useful if you have a script that you want to run with a shell other than the shell you are running the script from. The following example shows a script that should be executed by tcsh:

$ cat tcsh_script
#!/bin/tcsh
echo "This is a tcsh script."
set person = jenny
echo "person is $person"

Because of the #! line, the operating system ensures that tcsh executes the script no matter which shell you run it from.

You can use ps -l within a shell script to display the name of the shell that is executing the script. The three lines that ps displays in the following example show the process running the parent bash shell, the process running the tcsh script, and the process running the ps command:

$ cat tcsh_script2
#!/bin/tcsh
ps -l

$ tcsh_script2
  UID  PID PPID CPU PRI NI   VSZ  RSS WCHAN STAT TT    TIME COMMAND
  501  915  914   0  31  0 27792  828 -     S+   p2 0:00.07 -bash
  501 2160 2156   0  31  0 27792  808 -     Ss   p4 0:00.02 -bash
  501 2165 2160   0  31  0 31816 1148 -     S+   p4 0:00.01 /bin/tc

If you do not follow #! with the name of an executable program, the shell reports that it cannot find the command that you asked it to run. You can optionally follow #! with SPACEs. If you omit the #! line and try to run, for example, a tcsh script from bash, the shell may generate error messages or the script may not run properly.

See page 653 for an example of a stand-alone sed script that uses #!.

# Begins a Comment

Comments make shell scripts and all code easier to read and maintain by you and others. The comment syntax is common to both the Bourne Again and the TC Shells.

If a pound sign (#) in the first character position of the first line of a script is not immediately followed by an exclamation point (!) or if a pound sign occurs in any other location in a script, the shell interprets it as the beginning of a comment. The shell then ignores everything between the pound sign and the end of the line (the next NEWLINE character).

Running a Shell Script

fork and exec system calls

A command on the command line causes the shell to fork a new process, creating a duplicate of the shell process (a subshell). The new process attempts to exec (execute) the command. Like fork, the exec routine is executed by the operating system (a system call). If the command is a binary executable program, such as a compiled C program, exec succeeds and the system overlays the newly created subshell with the executable program. If the command is a shell script, exec fails. When exec fails, the command is assumed to be a shell script, and the subshell runs the commands in the script. Unlike a login shell, which expects input from the command line, the subshell takes its input from a file: the shell script.

As discussed earlier, if you have a shell script in a file that you do not have execute permission for, you can run the commands in the script by using a bash command to exec a shell to run the script directly. In the following example, bash creates a new shell that takes its input from the file named whoson:

$ bash whoson

Because the bash command expects to read a file containing commands, you do not need execute permission for whoson. (You do need read permission.) Even though bash reads and executes the commands in whoson, standard input, standard output, and standard error remain connected to the terminal.

Although you can use bash to execute a shell script, this technique causes the script to run more slowly than giving yourself execute permission and directly invoking the script. Users typically prefer to make the file executable and run the script by typing its name on the command line. It is also easier to type the name, and this practice is consistent with the way other kinds of programs are invoked (so you do not need to know whether you are running a shell script or another kind of program). However, if bash is not your interactive shell or if you want to see how the script runs with different shells, you may want to run a script as an argument to bash or tcsh.

Separating and Grouping Commands

Whether you give the shell commands interactively or write a shell script, you must separate commands from one another. This section, which applies to the Bourne Again and the TC Shells, reviews the ways to separate commands that were covered in Chapter 5 and introduces a few new ones.

Caution: sh does not call the original Bourne Shell

The original Bourne Shell was invoked with the command sh. Although you can call bash with an sh command, it is not the original Bourne Shell. The sh command (/bin/sh) is a copy of /bin/bash, so it is simply another name for the bash command. When you call bash using the command sh, bash TRies to mimic the behavior of the original Bourne Shell as closely as possible. It does not always succeed.

; and NEWLINE Separate Commands

The NEWLINE character is a unique command separator because it initiates execution of the command preceding it. You have seen this throughout this book each time you press the RETURN key at the end of a command line.

The semicolon (;) is a command separator that does not initiate execution of a command and does not change any aspect of how the command functions. You can execute a series of commands sequentially by entering them on a single command line and separating each from the next with a semicolon (;). You initiate execution of the sequence of commands by pressing RETURN:

$ x ; y ; z

If x, y, and z are commands, the preceding command line yields the same results as the next three commands. The difference is that in the next example the shell issues a prompt after each of the commands (x, y, and z) finishes executing, whereas the preceding command line causes the shell to issue a prompt only after z is complete:

$ x
$ y
$ z

Whitespace

Although the whitespace around the semicolons in the earlier example makes the command line easier to read, it is not necessary. None of the command separators needs to be surrounded by SPACEs or TABs.

\ Continues a Command

When you enter a long command line and the cursor reaches the right side of the screen, you can use a backslash (\) character to continue the command on the next line. The backslash quotes, or escapes, the NEWLINE character that follows it so that the shell does not treat the NEWLINE as a command terminator. Enclosing a backslash within single quotation marks turns off the power of a backslash to quote special characters such as NEWLINE. Enclosing a backslash within double quotation marks has no effect on the power of the backslash.

Although you can break a line in the middle of a word (token), it is typically easier to break a line just before or after whitespace.

Optional

You can enter a RETURN in the middle of a quoted string on a command line without using a backslash. The NEWLINE (RETURN) that you enter will then be part of the string:

$ echo "Please enter the three values
> required to complete the transaction."
Please enter the three values
required to complete the transaction.

In the three examples in this section, the shell does not interpret RETURN as a command terminator because it occurs within a quoted string. The > is a secondary prompt indicating that the shell is waiting for you to continue the unfinished command. In the next example, the first RETURN is quoted (escaped) so the shell treats it as a separator and does not interpret it literally.

$ echo "Please enter the three values \
> required to complete the transaction."
Please enter the three values required to complete the transaction.

Single quotation marks cause the shell to interpret a backslash literally:

$ echo 'Please enter the three values \
> required to complete the transaction.'
Please enter the three values \
required to complete the transaction.

| and & Separate Commands and Do Something Else

The pipe symbol (|) and the background task symbol (&) are also command separators. They do not start execution of a command but do change some aspect of how the command functions. The pipe symbol alters the source of standard input or the destination of standard output. The background task symbol causes the shell to execute the task in the background so you get a prompt immediately and can continue working on other tasks.

Each of the following command lines initiates a single job comprising three tasks:

$ x | y | z
$ ls -l | grep tmp | less

In the first job, the shell redirects standard output of task x to standard input of task y and redirects y's standard output to z's standard input. Because it runs the entire job in the foreground, the shell does not display a prompt until task z runs to completion: Task z does not finish until task y finishes, and task y does not finish until task x finishes. In the second job, task x is an ls -l command, task y is grep tmp, and task z is the pager less. The shell displays a long (wide) listing of the files in the working directory that contain the string tmp, piped through less.

The next command line executes tasks d and e in the background and task f in the foreground:

$ d & e & f
[1] 14271
[2] 14272

The shell displays the job number between brackets and the PID (process identification) number for each process running in the background. You get a prompt as soon as f finishes, which may be before d or e finishes.

Before displaying a prompt for a new command, the shell checks whether any background jobs have completed. For each job that has completed, the shell displays its job number, the word Done, and the command line that invoked the job; then the shell displays a prompt. When the job numbers are listed, the number of the last job started is followed by a + character and the job number of the previous job is followed by a - character. Any other jobs listed show a SPACE character. After running the last command, the shell displays the following before issuing a prompt:

[1]- Done          d
[2]+ Done          e

The next command line executes all three tasks as background jobs. You get a shell prompt immediately:

$ d & e & f &
[1] 14290
[2] 14291
[3] 14292

You can use pipes to send the output from one task to the next task and an ampersand (&) to run the entire job as a background task. Again the prompt comes back immediately. The shell regards the commands joined by a pipe as being a single job. That is, it treats all pipes as single jobs, no matter how many tasks are connected with the pipe (|) symbol or how complex they are. The Bourne Again Shell shows only one process placed in the background:

$ d e f &
[1] 14295

The TC Shell shows three processes (all belonging to job 1) placed in the background:

tcsh $ d | e | f &
[1] 14302 14304 14306

Optional: () Groups Commands

You can use parentheses to group commands. The shell creates a copy of itself, called a subshell, for each group. It treats each group of commands as a job and creates a new process to execute each command (refer to "Process Structure" on page 293 for more information on creating subshells). Each subshell (job) has its own environment, meaning that it has its own set of variables with values that can differ from those of other subshells.

The following command line executes commands a and b sequentially in the background while executing c in the background. The shell prompt returns immediately.

$ (a ; b) & c &
[1] 15520
[2] 15521

The preceding example differs from the earlier example d & e & f & in that tasks a and b are initiated sequentially, not concurrently.

Similarly the following command line executes a and b sequentially in the background and, at the same time, executes c and d sequentially in the background. The subshell running a and b and the subshell running c and d run concurrently. The prompt returns immediately.

$ (a ; b) & (c ; d) &
[1] 15528
[2] 15529

The next script copies one directory to another. The second pair of parentheses creates a subshell to run the commands following the pipe. Because of these parentheses, the output of the first tar command is available for the second tar command despite the intervening cd command. Without the parentheses, the output of the first tar command would be sent to cd and lost because cd does not process input from standard input. The shell variables $1 and $2 represent the first and second command line arguments (page 565), respectively. The first pair of parentheses, which creates a subshell to run the first two commands, allows users to call cpdir with relative pathnames. Without them the first cd command would change the working directory of the script (and consequently the working directory of the second cd command). With them only the working directory of the subshell is changed.

$ cat cpdir
(cd $1 ; tar -cf - . ) | (cd $2 ; tar -xvf - )
$ cpdir /Users/alex/sources /Users/alex/memo/biblio

The cpdir command line copies the files and directories in the /Users/alex/sources directory to the directory named /Users/alex/memo/biblio. This shell script is almost the same as using cp with the -r option. Refer to Part V for more information on cp (page 690) and tar (page 862).

Job Control

A job is a command pipeline. You run a simple job whenever you give the shell a command. For example, type date on the command line and press RETURN: You have run a job. You can also create several jobs with multiple commands on a single command line:

$ find . -print  | sort | lpr & grep -l alex /tmp/* > alexfiles &
[1] 18839
[2] 18876

The portion of the command line up to the first & is one job consisting of three processes connected by pipes: find (page 728), sort (page 49), and lpr (page 45). The second job is a single process running grep. Both jobs have been put into the background by the trailing & characters, so bash does not wait for them to complete before displaying a prompt.

Using job control you can move commands from the foreground to the background (and vice versa), stop commands temporarily, and list all the commands that are running in the background or stopped.

jobs: Lists Jobs

The jobs builtin lists all background jobs. The following sequence demonstrates what happens when you give a jobs command. Here the sleep command runs in the background and creates a background job that jobs reports on:

$ sleep 60 &
[1] 7809
$ jobs
[1] + Running                           sleep 60 &

fg: Brings a Job to the Foreground

The shell assigns job numbers to commands you run in the background (page 270). Several jobs are started in the background in the next example. For each job the shell lists the job number and PID number immediately, just before it issues a prompt.

$ vim memo &
[1] 1246
$ date &
[2] 1247
$ Sun Dec 4 11:44:40 PST 2005
[2]+ Done          date
$ find /usr -name ace -print > findout &
[2] 1269
$ jobs
[1]- Running        vim memo &
[2]+ Running        find /usr -name ace -print > findout &

Job numbers, which are discarded when a job is finished, can be reused. When you start or put a job in the background, the shell assigns a job number that is one more than the highest job number in use.

In the preceding example, the jobs command lists the first job, vim memo, as job 1. The date command does not appear in the jobs list because it finished before jobs was run. Because the date command was completed before find was run, the find command became job 2.

To move a background job into the foreground, use the fg builtin followed by the job number. Alternatively, you can give a percent sign (%) followed immediately by the job number as a command. Either of the following commands moves job 2 into the foreground:

$ fg 2

or

$ %2

You can also refer to a job by following the percent sign with a string that uniquely identifies the beginning of the command line used to start the job. Instead of the preceding command, you could have used either fg %find or fg %f because both uniquely identify job 2. If you follow the percent sign with a question mark and a string, the string can match any part of the command line. In the preceding example, fg %?ace also brings job 2 into the foreground.

Often the job you wish to bring into the foreground is the only job running in the background or is the job that jobs lists with a plus (+). In these cases you can use fg without an argument.

bg: Sends a Job to the Background

To move the foreground job to the background, you must first suspend (temporarily stop) the job by pressing the suspend key (usually CONTROL-Z). Pressing the suspend key immediately suspends the job in the foreground. You can then use the bg builtin to resume execution of the job in the background.

$ bg

If a background job attempts to read from the terminal, the shell stops it and notifies you that the job has been stopped and is waiting for input. You must then move the job into the foreground so that it can read from the terminal. The shell displays the command line when it moves the job into the foreground.

$ (sleep 5; cat > mytext) &
[1] 1343
$ date
Sun Dec 4 11:58:20 PST 2005
[1]+ Stopped                 ( sleep 5; cat >mytext )
$ fg
( sleep 5; cat >mytext )
Remember to let the cat out!
CONTROL-D
$

In the preceding example, the shell displays the job number and PID number of the background job as soon as it starts, followed by a prompt. Demonstrating that you can give a command at this point, the user gives the command date and its output appears on the screen. The shell waits until just before it issues a prompt (after date has finished) to notify you that job 1 is stopped. When you give an fg command, the shell puts the job in the foreground and you can enter the input that the command is waiting for. In this case the input needs to be terminated with a CONTROL-D to signify EOF (end of file). The shell then displays another prompt.

The shell keeps you informed about changes in the status of a job, notifying you when a background job starts, completes, or is stopped, perhaps waiting for input from the terminal. The shell also lets you know when a foreground job is suspended. Because notices about a job being run in the background can disrupt your work, the shell delays displaying these notices until just before it displays a prompt. You can set notify (page 320) to make the shell display these notices without delay.

If you try to exit from a shell while jobs are stopped, the shell issues a warning and does not allow you to exit. If you then use jobs to review the list of jobs or you immediately try to leave the shell again, the shell allows you to leave and terminates the stopped jobs. Jobs that are running (not stopped) in the background continue to run. In the following example, find (job 1) continues to run after the second exit terminates the shell, but cat (job 2) is terminated:

$ find / -size +100k > $HOME/bigfiles 2>&1 &
[1] 1426
$ cat > mytest &
[2] 1428
$ exit
exit
There are stopped jobs.

[2]+ Stopped                 cat >mytest
$ exit
exit
login:

Manipulating the Directory Stack

Both the Bourne Again and the TC Shells allow you to store a list of directories you are working with, enabling you to move easily among them. This list is referred to as a stack. It is analogous to a stack of dinner plates: You typically add plates to and remove plates from the top of the stack, creating a last-in first-out, (LIFO) stack.

dirs: Displays the Stack

The dirs builtin displays the contents of the directory stack. If you call dirs when the directory stack is empty, it displays the name of the working directory:

$ dirs
~/literature

The dirs builtin uses a tilde (~) to represent the name of the home directory. The examples in the next several sections assume that you are referring to the directory structure shown in Figure 8-2.

Figure 8-2. The directory structure in the examples

pushd: Pushes a Directory on the Stack

To change directories and at the same time add a new directory to the top of the stack, use the pushd (push directory) builtin. In addition to changing directories, the pushd builtin displays the contents of the stack. The following example is illustrated in Figure 8-3:

$ pushd ../demo
~/demo ~/literature
$ pwd
/Users/sam/demo
$ pushd ../names
~/names ~/demo ~/literature
$ pwd
/Users/sam/names

Figure 8-3. Creating a directory stack

When you use pushd without an argument, it swaps the top two directories on the stack and makes the new top directory (which was the second directory) become the new working directory (Figure 8-4):

$ pushd
~/demo ~/names ~/literature
$ pwd
/Users/sam/demo

Figure 8-4. Using pushd to change working directories

Using pushd in this way, you can easily move back and forth between two directories. You can also use cd - to change to the previous directory, whether or not you have explicitly created a directory stack. To access another directory in the stack, call pushd with a numeric argument preceded by a plus sign. The directories in the stack are numbered starting with the top directory, which is number 0. The following pushd command continues with the previous example, changing the working directory to literature and moving literature to the top of the stack:

$ pushd +2
~/literature ~/demo ~/names
$ pwd
/Users/sam/literature

popd: Pops a Directory Off the Stack

To remove a directory from the stack, use the popd (pop directory) builtin. As the following example and Figure 8-5 show, popd used without an argument removes the top directory from the stack and changes the working directory to the new top directory:

$ dirs
~/literature ~/demo ~/names
$ popd
~/demo ~/names
$ pwd
/Users/sam/demo

Figure 8-5. Using popd to remove a directory from the stack

To remove a directory other than the top one from the stack, use popd with a numeric argument preceded by a plus sign. The following example removes directory number 1, demo:

$ dirs
~/literature ~/demo ~/names
$ popd +1
~/literature ~/names

Removing a directory other than directory number 0 does not change the working directory.


Parameters and Variables

Variables

Within a shell, a shell parameter is associated with a value that is accessible to the user. There are several kinds of shell parameters. Parameters whose names consist of letters, digits, and underscores are often referred to as shell variables, or simply variables. A variable name must start with a letter or underscore, not with a number. Thus A76, MY_CAT, and ___X___ are valid variable names, whereas 69TH_STREET (starts with a digit) and MY-NAME (contains a hyphen) are not.

User-created variables

Shell variables that you name and assign values to are user-created variables. You can change the values of user-created variables at any time, or you can make them readonly so that their values cannot be changed. You can also make user-created variables global. A global variable (also called an environment variable) is available to all shells and other programs you fork from the original shell. One naming convention is to use only uppercase letters for global variables and to use mixed-case or lowercase letters for other variables. Refer to "Locality of Variables" on page 560 for more information on global variables.

To assign a value to a variable in the Bourne Again Shell, use the following syntax:

VARIABLE=value

There can be no whitespace on either side of the equal sign (=). An example assignment follows:

$ myvar=abc

Under the TC Shell the assignment must be preceded by the word set and the SPACEs on either side of the equal sign are optional:

$ set myvar = abc

The Bourne Again Shell permits you to put variable assignments on a command line. These assignments are local to the command shellthat is, they apply to the command only. The my_script shell script displays the value of TEMPDIR. The following command runs my_script with TEMPDIR set to /Users/sam/temp. The echo builtin shows that the interactive shell has no value for TEMPDIR after running my_script. If TEMPDIR had been set in the interactive shell, running my_script in this manner would have had no effect on its value.

$ cat my_script
echo $TEMPDIR
$ TEMPDIR=/Users/sam/temp my_script
/Users/sam/temp
$ echo $TEMPDIR

$

Keyword variables

Keyword shell variables (or simply keyword variables) have special meaning to the shell and usually have short, mnemonic names. When you start a shell (by logging in, for example), the shell inherits several keyword variables from the environment. Among these variables are HOME, which identifies your home directory, and PATH, which determines which directories the shell searches and in what order to locate commands that you give the shell. The shell creates and initializes (with default values) other keyword variables when you start it. Still other variables do not exist until you set them.

You can change the values of most of the keyword shell variables at any time but it is usually not necessary to change the values of keyword variables initialized in the /etc/profile or /etc/csh.cshrc systemwide startup files. If you need to change the value of a bash keyword variable, do so in one of your startup files (for bash see page 259; for tcsh see page 340). Just as you can make user-created variables global, so you can make keyword variables global; this is usually done automatically in the startup files. You can also make a keyword variable readonly.

Positional parameters Special parameters

The names of one group of parameters do not resemble variable names. Most of these parameters have one-character names (for example, 1, ?, and #) and are referenced (as are all variables) by preceding the name with a dollar sign ($1, $?, and $#). The values of these parameters reflect different aspects of your ongoing interaction with the shell.

Whenever you give a command, each argument on the command line becomes the value of a positional parameter. Positional parameters (page 564) enable you to access command line arguments, a capability that you will often require when you write shell scripts. The set builtin (page 568) enables you to assign values to positional parameters.

Other frequently needed shell script values, such as the name of the last command executed, the number of command line arguments, and the status of the most recently executed command, are available as special parameters. You cannot assign values to special parameters.

User-Created Variables

The first line in the following example declares the variable named person and initializes it with the value alex (use set person = alex in tcsh):

$ person=alex
$ echo person
person
$ echo $person
alex

Because the echo builtin copies its arguments to standard output, you can use it to display the values of variables. The second line of the preceding example shows that person does not represent alex. Instead, the string person is echoed as person. The shell substitutes the value of a variable only when you precede the name of the variable with a dollar sign ($). The command echo $person displays the value of the variable person; it does not display $person because the shell does not pass $person to echo as an argument. Because of the leading $, the shell recognizes that $person is the name of a variable, substitutes the value of the variable, and passes that value to echo. The echo builtin displays the value of the variablenot its namenever knowing that you called it with a variable.

Quoting the $

You can prevent the shell from substituting the value of a variable by quoting the leading $. Double quotation marks do not prevent the substitution; single quotation marks or a backslash (\) do.

$ echo $person
alex
$ echo "$person"
alex
$ echo '$person'
$person
$ echo \$person
$person

SPACEs

Because they do not prevent variable substitution but do turn off the special meanings of most other characters, double quotation marks are useful when you assign values to variables and when you use those values. To assign a value that contains SPACEs or TABs to a variable, use double quotation marks around the value. Although double quotation marks are not required in all cases, using them is a good habit.

$ person="alex and jenny"
$ echo $person
alex and jenny
$ person=alex and jenny
bash: and: command not found

When you reference a variable that contains TABs or multiple adjacent SPACEs, you need to use quotation marks to preserve the spacing. If you do not quote the variable, the shell collapses each string of blank characters into a single SPACE before passing the variable to the utility:

$ person="alex   and  jenny"
$ echo $person
alex and jenny
$ echo "$person"
alex and jenny

When you execute a command with a variable as an argument, the shell replaces the name of the variable with the value of the variable and passes that value to the program being executed. If the value of the variable contains a special character, such as * or ?, the shell may expand that variable.

Pathname expansion in assignments

The first line in the following sequence of commands assigns the string alex * to the variable memo. The Bourne Again Shell does not expand the string because bash does not perform pathname expansion (page 133) when assigning a value to a variable. All shells process a command line in a specific order. Within this order bash (but not tcsh) expands variables before it interprets commands. In the following echo command line, the double quotation marks quote the asterisk (*) in the expanded value of $memo and prevent bash from performing pathname expansion on the expanded memo variable before passing its value to the echo command:

$ memo=alex*
$ echo "$memo"
alex*

All shells interpret special characters as special when you reference a variable that contains an unquoted special character. In the following example, the shell expands the value of the memo variable because it is not quoted:

$ ls
alex.report
alex.summary
$ echo $memo
alex.report alex.summary

Here the shell expands $memo to alex*, expands alex* to alex.report and alex.summary, and passes these two values to echo.

Optional

Braces

The $VARIABLE syntax is a special case of the more general syntax ${VARIABLE}, in which the variable name is enclosed by ${}. The braces insulate the variable name. Braces are necessary when catenating a variable value with a string:

$ PREF=counter
$ WAY=$PREFclockwise
$ FAKE=$PREFfeit
$ echo $WAY $FAKE

$

The preceding example does not work as planned. Only a blank line is output because, although the symbols PREFclockwise and PREFfeit are valid variable names, they are not set. By default the shell evaluates an unset variable as an empty (null) string and displays this value (bash) or generates an error message (tcsh). To achieve the intent of these statements, refer to the PREF variable using braces:

$ PREF=counter
$ WAY=${PREF}clockwise
$ FAKE=${PREF}feit
$ echo $WAY $FAKE
counterclockwise counterfeit

The Bourne Again Shell refers to the arguments on its command line by position, using the special variables $1, $2, $3, and so forth up to $9. If you wish to refer to arguments past the ninth argument, you must use braces: ${10}. The name of the command is held in $0 (page 565).

unset: Removes a Variable

Unless you remove a variable, it exists as long as the shell in which it was created exists. To remove the value of a variable but not the variable itself, set the value to null (use set person = in tcsh):

$ person=
$ echo $person
$

You can remove a variable with the unset builtin. To remove the variable person, give the following command:

$ unset person

Variable Attributes

This section discusses attributes and explains how to assign them to variables.

readonly: Makes the Value of a Variable Permanent

You can use the readonly builtin (not in tcsh) to ensure that the value of a variable cannot be changed. The next example declares the variable person to be readonly. You must assign a value to a variable before you declare it to be readonly; you cannot change its value after the declaration. When you attempt to unset or change the value of a readonly variable, the shell displays an error message:

$ person=jenny
$ echo $person
jenny
$ readonly person
$ person=helen
bash: person: readonly variable

If you use the readonly builtin without an argument, it displays a list of all readonly shell variables. This list includes keyword variables that are automatically set as readonly as well as keyword or user-created variables that you have declared as readonly. See "Listing variable attributes" on page 283 for an example (readonly and declare -r produce the same output).

declare and typeset: Assign Attributes to Variables

The declare and typeset builtins (two names for the same command, neither of which is available in tcsh) set attributes and values for shell variables. Table 8-3 lists five of these attributes.

Table 8-3. Variable attributes (typeset or declare)

Attribute

Meaning

-a

Declares a variable as an array (page 558)

-f

Declares a variable to be a function name (page 314)

-i

Declares a variable to be of type integer (page 284)

-r

Makes a variable readonly; also readonly (page 282)

-x

Exports a variable (makes it global); also export (page 560)

The following commands declare several variables and set some attributes. The first line declares person1 and assigns it a value of alex. This command has the same effect with or without the word declare.

$ declare person1=alex
$ declare -r person2=jenny
$ declare -rx person3=helen
$ declare -x person4

The readonly and export builtins are synonyms for the commands declare -r and declare -x, respectively. It is legal to declare a variable without assigning a value to it, as the preceding declaration of the variable person4 illustrates. This declaration makes person4 available to all subshells (makes it global). Until an assignment is made to the variable, it has a null value.

You can list the options to declare separately in any order. The following is equivalent to the preceding declaration of person3:

$ declare -x -r person3=helen

Use the + character in place of - when you want to remove an attribute from a variable. You cannot remove a readonly attribute however. After the following command is given, the variable person3 is no longer exported but it is still readonly.

$ declare +x person3

You can also use typeset instead of declare.

Listing variable attributes

Without any arguments or options, the declare builtin lists all shell variables. The same list is output when you run set (page 568) without any arguments.

If you use a declare builtin with options but no variable names as arguments, the command lists all shell variables that have the indicated attributes set. For example, the option -r with declare gives a list of all readonly shell variables. This list is the same as that produced by a readonly command without any arguments. After the declarations in the preceding example have been given, the results are as follows:

$ declare -r
declare -ar BASH_VERSINFO='([0]="2" [1]="05b" [2]="0" [3]="1" ... )'
declare -ir EUID="500"
declare -ir PPID="936"
declare -r SHELLOPTS="braceexpand:emacs:hashall:histexpand:history:..."
declare -ir UID="500"
declare -r person2="jenny"
declare -rx person3="helen"

The first five entries are keyword variables that are automatically declared as readonly. Some of these variables are stored as integers (-i). The -a option indicates that BASH_VERSINFO is an array variable; the value of each element of the array is listed to the right of an equal sign.

Integer

By default the values of variables are stored as strings. When you perform arithmetic on a string variable, the shell converts the variable into a number, manipulates it, and then converts it back to a string. A variable with the integer attribute is stored as an integer. Assign the integer attribute as follows:

$ typeset -i COUNT

Keyword Variables

Keyword variables either are inherited or are declared and initialized by the shell when it starts. You can assign values to these variables from the command line or from a startup file. Typically you want these variables to apply to all subshells you start as well as to your login shell. For those variables not automatically exported by the shell, you must use export (bash, page 560) or setenv (tcsh, page 353) to make them available to child shells.

HOME: Your Home Directory

By default your home directory is your working directory when you log in. Your home directory is determined when you establish your account; its name is stored in the NetInfo database (page 441).

When you log in, the shell inherits the pathname of your home directory and assigns it to the variable HOME. When you give a cd command without an argument, cd makes the directory whose name is stored in HOME the working directory:

$ pwd
/Users/alex/laptop
$ echo $HOME
/Users/alex
$ cd
$ pwd
/Users/alex

This example shows the value of the HOME variable and the effect of the cd builtin. After you execute cd without an argument, the pathname of the working directory is the same as the value of HOME: your home directory.

Tilde (~)

The shell uses the value of HOME to expand pathnames that use the shorthand tilde (~) notation (page 78) to denote a user's home directory. The following example uses echo to display the value of this shortcut and then uses ls to list the files in Alex's laptop directory, which is a subdirectory of his home directory:

$ echo ~
/Users/alex
$ ls ~/laptop
tester     count     lineup

PATH: Where the Shell Looks for Programs

When you give the shell an absolute or relative pathname rather than a simple filename as a command, it looks in the specified directory for an executable file with the specified filename. If the file with the pathname you specified does not exist, the shell reports command not found. If the file exists as specified but you do not have execute permission for it, or in the case of a shell script you do not have read and execute permission for it, the shell reports Permission denied.

If you give a simple filename as a command, the shell searches through certain directories for the program you want to execute. It looks in several directories for a file that has the same name as the command and that you have execute permission for (a compiled program) or read and execute permission for (a shell script). The PATH shell variable controls this search.

The default value of PATH is determined when bash or tcsh is compiled. It is not set in a startup file, although it may be modified there. Normally the default specifies that the shell search several system directories used to hold common commands and then search the working directory. These system directories include /bin and /usr/bin and other directories appropriate to the local system. When you give a command, if the shell does not find the executableand, in the case of a shell script, readablefile named by the command in any of the directories listed in PATH, the shell generates one of the aforementioned error messages.

Working directory

The PATH variable specifies the directories in the order the shell should search them. Each directory must be separated from the next by a colon. The following command sets PATH so that a search for an executable file starts with the /usr/local/bin directory. If it does not find the file in this directory, the shell first looks in /bin, and then in /usr/bin. If the search fails in those directories, the shell looks in the bin director, a subdirectory of the user's home directory. Finally the shell looks in the working directory. Exporting PATH makes its value accessible to subshells:

$ export PATH=/usr/local/bin:/bin:/usr/bin:~/bin:

A null value in the string indicates the working directory. In the preceding example, a null value (nothing between the colon and the end of the line) appears as the last element of the string. The working directory is represented by a leading colon (not recommended; see the following security tip), a trailing colon (as in the example), or two colons next to each other anywhere in the string. You can also represent the working directory explicitly with a period (.).

See "PATH" on page 359 for a tcsh example. Because Mac OS X stores many executable files in directories named bin (binary), users typically put their own executable files in their own ~/bin directories. If you put your own bin directory at the end of your PATH, as in the preceding example, the shell looks there for any commands that it cannot find in directories listed earlier in PATH.

secutiry: PATH and security

Do not put the working directory first in PATH when security is a concern. If you are running as Superuser, you should never put the working directory first in PATH. It is common for Superuser PATH to omit the working directory entirely. You can always execute a file in the working directory by prepending ./ to the name: ./ls.

Putting the working directory first in PATH can create a security hole. Most people type ls as the first command when entering a directory. If the owner of a directory places an executable file named ls in the directory, and the working directory appears first in a user's PATH, the user giving an ls command from the directory executes the ls program in the working directory instead of the system ls utility, possibly with undesirable results.

If you want to add directories to PATH, you can reference the old value of the PATH variable while you are setting PATH to a new value (but see the preceding security tip). The following command adds /usr/X11R6/bin to the beginning of the current PATH and /usr/local/bin and the working directory to the end:

$ PATH=/usr/X11R6/bin:$PATH:/usr/local/bin

MAIL: Where Your Mail Is Kept

The MAIL variable (mail under tcsh) contains the pathname of the file that holds your mail (your mailbox, usually /var/mail/name, where name is your login name). If MAIL is set and MAILPATH (next) is not set, the shell informs you when mail arrives in the file specified by MAIL. In a graphical environment you can unset MAIL so that the shell does not display mail reminders in a terminal emulator window (assuming you are using a graphical mail program). Most Mac OS X systems do not use local files for incoming mail; mail is typically kept on a remote mail server instead. The MAIL variable and other mail-related shell variables do not do anything unless you have a local mail server.

The MAILPATH variable (not available under tcsh) contains a list of filenames separated by colons. If this variable is set, the shell informs you when any one of the files is modified (for example, when mail arrives). You can follow any of the filenames in the list with a question mark (?), followed by a message. The message replaces the you have mail message when you get mail while you are logged in.

The MAILCHECK variable (not available under tcsh) specifies how often, in seconds, the shell checks for new mail. The default is 60 seconds. If you set this variable to zero, the shell checks before each prompt.

PS1: User Prompt (Primary)

The default Bourne Again Shell prompt is a dollar sign ($). When you run bash as root, you may have a pound sign (#) prompt. The PS1 variable (prompt under tcsh, page 360) holds the prompt string that the shell uses to let you know that it is waiting for a command. When you change the value of PS1 or prompt, you change the appearance of your prompt.

You can customize the prompt displayed by PS1. For example, the assignment

$ PS1="[\u@\h \W \!]$ "

displays the following prompt:

[user@host directory event]$

where user is the username, host is the hostname up to the first period, directory is the basename of the working directory, and event is the event number of the current command.

If you are working on more than one system, it can be helpful to incorporate the system name into your prompt. For example, you might change the prompt to the name of the system you are using, followed by a colon and a SPACE (a SPACE at the end of the prompt makes the commands that you enter after the prompt easier to read):

$ PS1="$(hostname): "
bravo.example.com: echo test
test
bravo.example.com:

Use the following command under tcsh:

tcsh $ set prompt = "`hostname` : "

The first example that follows changes the prompt to the name of the local host, a SPACE, and a dollar sign (or, if the user is running as root, a pound sign). The second example changes the prompt to the time followed by the name of the user. The third example changes the prompt to the one used in this book (a pound sign for root and a dollar sign otherwise):

$ PS1='\h \$ '
bravo $

$ PS1='\@ \u $ '
09:44 PM alex $

$ PS1='\$ '
$

Table 8-4 describes some of the symbols you can use in PS1. For a complete list of special characters you can use in the prompt strings, open the bash man page and search for the second occurrence of PROMPTING (give the command /PROMPTING and then press n).

Table 8-4. PS1 symbols

Symbol

Display in prompt

\$

# if the user is running as root; otherwise, $

\w

Pathname of the working directory

\W

Basename of the working directory

\!

Current event (history) number (page 299)

\d

Date in Weekday Month Date format

\h

Machine hostname, without the domain

\H

Full machine hostname, including the domain

\u

Username of the current user

\@

Current time of day in 12-hour, AM/PM format

\T

Current time of day in 12-hour HH:MM:SS format

\A

Current time of day in 24-hour HH:MM format

\t

Current time of day in 24-hour HH:MM:SS format

PS2: User Prompt (Secondary)

Prompt String 2 is a secondary prompt that the shell stores in PS2 (not under tcsh). On the first line of the next example, an unclosed quoted string follows echo. The shell assumes that the command is not finished and, on the second line, gives the default secondary prompt (>). This prompt indicates that the shell is waiting for the user to continue the command line. The shell waits until it receives the quotation mark that closes the string and then executes the command:

$ echo "demonstration of prompt string
> 2"
demonstration of prompt string
2
$ PS2="secondary prompt: "
$ echo "this demonstrates
secondary prompt: prompt string 2"
this demonstrates
prompt string 2

The second command changes the secondary prompt to secondary prompt: followed by a SPACE. A multiline echo demonstrates the new prompt.

PS3: Menu Prompt

PS3 holds the menu prompt for the select control structure (page 552).

PS4: Debugging Prompt

PS4 holds the bash debugging symbol (page 536).

IFS: Separates Input Fields (Word Splitting)

The IFS (Internal Field Separator) shell variable (not under tcsh) specifies the characters that you can use to separate arguments on a command line and has the default value of SPACE TAB NEWLINE. Regardless of the value of IFS, you can always use one or more SPACE or TAB characters to separate arguments on the command line, provided that these characters are not quoted or escaped. When you assign IFS character values, these characters can also separate fields but only if they undergo expansion. This type of interpretation of the command line is called word splitting.

Caution: Be careful when changing IFS

Changing IFS has a variety of side effects so work cautiously. You may find it useful to first save the value of IFS before changing it; you can easily then restore the original value if you get unexpected results. Alternatively, you can fork a new shell with a bash command before experimenting with IFS; if you get into trouble, you can exit back to the old shell, where IFS is working properly. You can also set IFS to its default value with the following command:

$ IFS=' \t\n'

The following example demonstrates how setting IFS can affect the interpretation of a command line:

$ a=w:x:y:z
$ cat $a
cat: w:x:y:z: No such file or directory
$ IFS=":"
$ cat $a
cat: w: No such file or directory
cat: x: No such file or directory
cat: y: No such file or directory
cat: z: No such file or directory

The first time cat is called, the shell expands the variable a, interpreting the string w:x:y:z as a single word to be used as the argument to cat. The cat utility cannot find a file named w:x:y:z and reports an error for that filename. After IFS is set to a colon (:), the shell expands the variable a into four words, each of which is an argument to cat. Now cat reports an error for four separate files: w, x, y, and z. Word splitting based on the colon (:) takes place only after the variable a is expanded.

The shell splits all expanded words on a command line according to the separating characters found in IFS. When there is no expansion, there is no splitting. Consider the following commands:

$ IFS="p"
$ export VAR

Although IFS is set to p, the p on the export command line is not expanded so the word export is not split.

The next example uses variable expansion in an attempt to produce an export command:

$ IFS="p"
$ aa=export
$ echo $aa
ex ort

This time expansion occurs so that the character p in the token export is interpreted as a separator as the preceding echo command shows. Now when you try to use the value of the aa variable to export the VAR variable, the shell parses the $aa VAR command line as ex ort VAR. The effect is that the command line starts the ex editor with two filenames: ort and VAR.

$ $aa VAR
2 files to edit
"ort" [New File]
Entering Ex mode.  Type "visual" to go to Normal mode.
:q
E173: 1 more file to edit
:q
$

If you unset IFS, only SPACEs and TABs work as field separators.

CDPATH: Broadens the Scope of cd

The CDPATH variable (cdpath under tcsh) allows you to use a simple filename as an argument to the cd builtin to change the working directory to a directory other than a child of the working directory. If you have several directories you like to work out of, this variable can speed things up and save you the tedium of using cd with longer pathnames to switch among them.

When CDPATH or cdpath is not set and you specify a simple filename as an argument to cd, cd searches the working directory for a subdirectory with the same name as the argument. If the subdirectory does not exist, cd displays an error message. When CDPATH or cdpath is set, cd searches for an appropriately named subdirectory in the directories in the CDPATH list. If cd finds one, that directory becomes the working directory. With CDPATH or cdpath set, you can use cd and a simple filename to change the working directory to a child of any of the directories listed in CDPATH or cdpath.

The CDPATH or cdpath variable takes on the value of a colon-separated list of directory pathnames (similar to the PATH variable). It is usually set in the ~/.bash_profile (bash) or ~/.tcshrc (tcsh) startup file with a command line such as the following:

export CDPATH=$HOME:$HOME/literature

Use the following format for tcsh:

setenv cdpath $HOME\:$HOME/literature

These commands cause cd to search your home directory, the literature directory, and then the working directory when you give a cd command. If you do not include the working directory in CDPATH or cdpath, cd searches the working directory if the search of all the other directories in CDPATH or cdpath fails. If you want cd to search the working directory first (which you should never do when you are logged in as rootrefer to the tip on page 286), include a null string, represented by two colons (::), as the first entry in CDPATH:

export CDPATH=::$HOME:$HOME/literature

If the argument to the cd builtin is an absolute pathnameone starting with a slash (/)the shell does not consult CDPATH or cdpath.

Keyword Variables: A Summary

Table 8-5 lists the bash keyword variables.

Table 8-5. bash keyword variables

Variable

Value

BASH_ENV

The pathname of the startup file for noninteractive shells (page 260)

CDPATH

The cd search path (page 290)

COLUMNS

The width of the display used by select (page 551)

FCEDIT

The name of the editor that fc uses by default (page 298)

HISTFILE

The pathname of the file that holds the history list (default: ~/.bash_history; page 295)

HISTFILESIZE

The maximum number of entries saved in HISTFILE (default: 500; page 295)

HISTSIZE

The maximum number of entries saved in the history list (default: 500; (page 295)

HOME

The pathname of the user's home directory (page 284); used as the default argument for cd and in tilde expansion (page 78)

IFS

Internal Field Separator (page 288); used for word splitting (page 328)

INPUTRC

The pathname of the Readline startup file (default: ~/.inputrc; page 308)

LANG

The locale category when that category is not specifically set with a LC_* variable

LC_*

A group of variables that specify locale categories including LC_COLLATE, LC_CTYPE, LC_MESSAGES, and LC_NUMERIC; use the locale builtin to display a complete list with values

LINES

The height of the display used by select (page 551)

MAIL

The pathname of the file that holds a user's mail (page 286)

MAILCHECK

How often, in seconds, bash checks for mail (page 286)

MAILPATH

A colon-separated list of file pathnames that bash checks for mail in (page 286)

PATH

A colon-separated list of directory pathnames that bash looks for commands in (page 285)

PROMPT_COMMAND

A command that bash executes just before it displays the primary prompt

PS1

Prompt String 1; the primary prompt (default: '\s\v\$ '; page 286)

PS2

Prompt String 2; the secondary prompt (default: '> '; page 288)

PS3

The prompt issued by select (page 551)

PS4

The bash debugging symbol (page 536)

REPLY

Holds the line that read accepts (page 572); also used by select (page 551)

Special Characters

Table 8-6 lists most of the characters that are special to the bash and tcsh shells.

Table 8-6. Shell special characters

Character

Use

NEWLINE

Initiates execution of a command (page 269)

;

Separates commands (page 269)

( )

Groups commands (page 271) for execution by a subshell or identifies a function (page 314)

&

Executes a command in the background (pages 131 and 270)

|

Sends output through a pipe (page 270)

>

Redirects standard output (page 122)

>>

Appends standard output (page 127)

<

Redirects standard input (page 124)

<<

Here document (page 553)

*

Any string of zero or more characters in an ambiguous file reference (page 134)

?

Any single character in an ambiguous file reference (page 134)

\

Quotes the following character (page 40)

'

Quotes a string, preventing all substitution (page 40)

"

Quotes a string, allowing only variable and command substitution (pages 40 and 280)

'...'

Performs command substitution (page 327)

[ ]

Character class in an ambiguous file reference (page 136)

$

References a variable (page 278)

. (dot builtin)

Executes a command (only at the beginning of a line, page 261)

#

Begins a comment (page 268)

{ }

Used to surround the contents of a function (page 314)

: (null builtin)

Returns true (page 579)

&& (Boolean AND)

Executes command on right only if command on left succeeds (returns a zero exit status, page 590)

|| (Boolean OR)

Executes command on right only if command on left fails (returns a nonzero exit status; page 590)

! (Boolean NOT)

Reverses exit status of a command

$() (not in tcsh)

Performs command substitution (preferred form; page 327)

[]

Evaluates an arithmetic expression (page 325)


Processes

A process is the execution of a command by Mac OS X. The shell that starts when you log in is a command, or a process, like any other. When you give the name of a Mac OS X utility on the command line, you initiate a process. When you run a shell script, another shell process is started and additional processes are created for each command in the script. Depending on how you invoke the shell script, the script is run either by the current shell or, more typically, by a subshell (child) of the current shell. A process is not started when you run a shell builtin, such as cd.

Process Structure

fork system call

Like the file structure, the process structure is hierarchical, with parents, children, and even a root. A parent process forks a child process, which in turn can fork other processes. (The term fork indicates that, as with a fork in the road, one process turns into two. Initially the two forks are identical except that one is identified as the parent and one as the child. You can also use the term spawn; the words are interchangeable.) The operating system routine, or system call, that creates a new process is named fork.

When Mac OS X begins execution when a system is started, it starts launchd, a single process called a spontaneous process, with PID number 1. This process holds the same position in the process structure as the root directory does in the file structure: It is the ancestor of all processes that the system and users work with. When the system is in multiuser mode, launchd runs additional processes that display the login window and may also display login: prompts on terminals or accept network logins. When a user responds to one of these prompts, the process that authenticates the user then starts a user environment such as the Finder for a graphical login, or the user's shell process, for a textual login.

Process Identification

PID number

Mac OS X assigns a unique PID (process identification) number at the inception of each process. As long as a process exists, it keeps the same PID number. During one session the same process is always executing the login shell. When you fork a new processfor example, when you use an editorthe PID number of the new (child) process is different from that of its parent process. When you return to the login shell, it is still being executed by the same process and has the same PID number as when you logged in.

The following example shows that the process running the shell forked (is the parent of) the process running ps (page 133). When you call it with the 1 option, ps displays a full listing of information about each process. The line of the ps display with bash in the COMMAND column refers to the process running the shell. The column headed by PID identifies the PID number. The column headed PPID identifies the PID number of the parent of the process. From the PID and PPID columns you can see that the process running the shell (PID 6476) is the parent of the process running sleep (PID 11040). The parent PID number of sleep is the same as the PID number of the shell (6476).

$ sleep 10 &
[1] 11040
$ ps -l
  UID   PID  PPID CPU PRI NI     VSZ   RSS WCHAN  STAT  TT       TIME COMMAND
  502  6476  6472   0  31  0   27800   864 -      Ss    p5    0:00.13 -bash
  502 11040  6476   0  31  0   27232   672 -      S     p5    0:00.01 sleep 10

Refer to page 819 for more information on ps and the columns it displays with the 1 option. A second pair of sleep and ps1 commands shows that the shell is still being run by the same process but that it forked another process to run sleep:

$ sleep 10 &
[1] 11046
$ ps -l
  UID   PID  PPID CPU PRI NI     VSZ   RSS WCHAN  STAT  TT       TIME COMMAND
  502  6476  6472   0  31  0   27800   864 -      Ss    p5    0:00.14 -bash
  502 11046  6476   0  31  0   27232   364 -      S     p5    0:00.00 sleep 10

Executing a Command

fork and sleep

When you give the shell a command, it usually forks (spawns) a child process to execute the command. While the child process is executing the command, the parent process sleeps. While a process is sleeping, it does not use any computer time but remains inactive, waiting to wake up. When the child process finishes executing the command, it tells its parent of its success or failure via its exit status and then dies. The parent process (which is running the shell) wakes up and prompts for another command.

Background process

When you run a process in the background by ending a command with an ampersand (&), the shell forks a child process without going to sleep and without waiting for the child process to run to completion. The parent process, which is executing the shell, reports the job number and PID number of the child and prompts for another command. The child process runs in the background, independent of its parent.

Builtins

Although the shell forks a process to run most of the commands you give it, some commands are built into the shell. The shell does not need to fork a process to run builtins. For more information refer to "Builtins" on page 138.

Variables

Within a given process, such as your login shell or a subshell, you can declare, initialize, read, and change variables. By default however, a variable is local to a process. When a process forks a child process, the parent does not pass the value of a variable to the child. You can make the value of a variable available to child processes (global) by using the export builtin under bash (page 560) or the setenv builtin under tcsh (page 353).


History

The history mechanism, a feature adapted from the C Shell, maintains a list of recently issued command lines, also called events, providing a quick way to reexecute any of the events in the list. This mechanism also enables you to execute variations of previous commands and to reuse arguments from them. You can replicate complicated commands and arguments that you used earlier in this login session or in a previous one and enter a series of commands that differ from one another in minor ways. The history list also serves as a record of what you have done. It can prove helpful when you have made a mistake and are not sure what you did or when you want to keep a record of a procedure that involved a series of commands.

The history builtin (both in bash and tcsh) displays the history list. If it does not, read onyou need to set some variables.

Tip: history can help track down mistakes

When you have made a command line mistake (not an error within a script or program) and are not sure what you did wrong, look at the history list to review your recent commands. Sometimes this list can help you figure out what went wrong and how to fix things.

Variables That Control History

The TC Shell's history mechanism is similar to bash's but uses different variables and has other differences. See page 342 for more information.

The value of the HISTSIZE variable determines the number of events preserved in the history list during a session. A value in the range of 100 to 1,000 is normal.

When you exit from the shell, the most recently executed commands are saved in the file given by the HISTFILE variable (the default is ~/.bash_history). The next time you start the shell, this file initializes the history list. The value of the HISTFILESIZE variable determines the number of lines of history saved in HISTFILE (not necessarily the same as HISTSIZE). HISTSIZE holds the number of events remembered during a session, HISTFILESIZE holds the number remembered between sessions, and the file designated by HISTFILE holds the history list. See Table 8-7.

Table 8-7. History variables

Function

Variable

Default

Maximum number of events saved during a session

HISTSIZE

500 events

Location of the history file

HISTFILE

~/.bash_history

Maximum number of events saved between sessions

HISTFILESIZE

500 events

Event number

The Bourne Again Shell assigns a sequential event number to each command line. You can display this event number as part of the bash prompt by including \! in PS1 (page 286). Examples in this section show numbered prompts when they help to illustrate the behavior of a command.

Give the following command manually or place it in ~/.bash_profile (to affect future sessions) to establish a history list of the 100 most recent events:

$ HISTSIZE=100

The following command causes bash to save the 100 most recent events across login sessions:

$ HISTFILESIZE=100

After you set HISTFILESIZE, you can log out and log in again, and the 100 most recent events from the previous login session will appear in your history list.

Give the command history to display the events in the history list. The list of events is ordered with oldest events at the top of the list. A tcsh history list includes the time the command was executed. The following history list includes a command to modify the bash prompt so that it displays the history event number. The last event in the history list is the history command that displayed the list.

32 $ history | tail
23 PS1="\! bash$ "
24 ls -l
25 cat temp
26 rm temp
27 vim memo
28 lpr memo
29 vim memo
30 lpr memo
31 rm memo
32 history | tail

As you run commands and your history list becomes longer, it may run off the top of the screen when you use the history builtin. Pipe the output of history tHRough less to browse through it, or give the command history 10 to look at the ten most recent commands.

Reexecuting and Editing Commands

You can reexecute any event in the history list. This feature can save you time, effort, and aggravation. Not having to reenter long command lines allows you to reexecute events more easily, quickly, and accurately than you could if you had to retype the entire command line. You can recall, modify, and reexecute previously executed events in three ways: You can use the fc builtin (covered next); the exclamation point commands page 299; or the Readline Library, which uses a one-line vi- or emacs-like editor to edit and execute events (page 304).

Tip: Which method to use?

If you are more familiar with vi or emacs and less familiar with the C or TC Shell, use fc or the Readline Library. If you are more familiar with the C or TC Shell and less familiar with vi and emacs, use the exclamation point commands. If it is a toss-up, try the Readline Library; it will benefit you in other areas of Mac OS X more than learning the exclamation point commands will.

fc: Displays, Edits, and Reexecutes Commands

The fc (fix command) builtin (not in tcsh) enables you to display the history list and to edit and reexecute previous commands. It provides many of the same capabilities as the command line editors.

Viewing the History List

When you call fc with the 1 option, it displays commands from the history list. Without any arguments, fc1 lists the 16 most recent commands in a numbered list, with the oldest appearing first:

$ fc -l
1024    cd
1025    view calendar
1026    vim letter.adams01
1027    vim letter.adams01
1028    lpr letter.adams01
1029    cd ../memos
1030    ls
1031    rm *0405
1032    fc -l
1033    cd
1034    pwd
1035    ls
...

The fc builtin can take zero, one, or two arguments with the 1 option. The arguments specify the part of the history list to be displayed:

fc l [ first [last]]

The fc builtin lists commands beginning with the most recent event that matches first. The argument can be an event number, the first few characters of the command line, or a negative number, which is taken to be the nth previous command. If you provide last, fc displays commands from the most recent event that matches first through the most recent event that matches last. The next command displays the history list from event 1027 through event 1030:

$ fc -l 1027 1030
1027     vim letter.adams01
1028     lpr letter.adams01
1029     cd ../memos
1030     ls

The following command lists the most recent event that begins with view through the most recent command line that begins with whereis:

$ fc -l view pwd
1025     view calendar
1026     vim letter.adams01
1027     vim letter.adams01
1028     lpr letter.adams01
1029     cd ../memos
1030     ls
1031     rm *0405
1032     fc -l
1033     cd
1034     pwd

To list a single command from the history list, use the same identifier for the first and second arguments. The following command lists event 1027:

$ fc -l 1027 1027
1027   vim letter.adams01

Editing and Reexecuting Previous Commands

You can use fc to edit and reexecute previous commands.

fc [e editor] [first [last]]

When you call fc with the e option followed by the name of an editor, fc calls the editor with event(s) in the Work buffer. Without first and last, fc defaults to the most recent command. The next example invokes the vi(m) editor to edit the most recent command:

$ fc -e vi

The fc builtin uses the stand-alone vi(m) editor. If you set the FCEDIT variable, you do not need to use the e option to specify an editor on the command line. Because the value of FCEDIT has been changed to /usr/bin/emacs and fc has no arguments, the following command edits the most recent command with the emacs editor:

$ export FCEDIT=/usr/bin/emacs
$ fc

If you call it with a single argument, fc invokes the editor on the specified command. The following example starts the editor with event 21 in the Work buffer. When you exit from the editor, the shell executes the command:

$ fc 21

Again you can identify commands with numbers or by specifying the first few characters of the command name. The following example calls the editor to work on events from the most recent event that begins with the letters vim through event 206:

$ fc vim 206

Caution: Clean up the fc buffer

When you execute an fc command, the shell executes whatever you leave in the editor buffer, possibly with unwanted results. If you decide you do not want to execute a command, delete everything from the buffer before you exit from the editor.

Reexecuting Commands Without Calling the Editor

You can reexecute previous commands without going into an editor. If you call fc with the s option, it skips the editing phase and reexecutes the command. The following example reexecutes event 1028:

$ fc -s 1028
lpr letter.adams01

The next example reexecutes the previous command:

$ fc -s

When you reexecute a command you can tell fc to substitute one string for another. The next example substitutes the string john for the string adams in event 1028 and executes the modified event:

$ fc -s adams=john 1028
lpr letter.john01

Using an Exclamation Point (!) to Reference Events

The C Shell history mechanism uses an exclamation point to reference events and is available under bash and tcsh. It is frequently more cumbersome to use than fc but nevertheless has some useful features. For example, the !! command reexecutes the previous event, and the !$ token represents the last word on the previous command line.

You can reference an event by using its absolute event number, its relative event number, or the text it contains. All references to events, called event designators, begin with an exclamation point (!). One or more characters follow the exclamation point to specify an event.

You can put history events anywhere on a command line. To escape an exclamation point so that it is treated literally instead of as the start of a history event, precede it with a backslash (\) or enclose it within single quotation marks.

Event Designators

An event designator specifies a command in the history list. See Table 8-8 on page 301 for a list of event designators.

Table 8-8. Event designators

Designator

Meaning

!

Starts a history event unless followed immediately by SPACE, NEWLINE, =, or (.

!!

The previous command.

!n

Command number n in the history list.

!n

The nth preceding command.

!string

The most recent command line that started with string.

!?string[?]

The most recent command that contained string. The last ? is optional.

!#

The current command (as you have it typed so far).

!{event}

The event is an event designator. The braces isolate event from the surrounding text. For example, !{3}3 is the third most recently executed command followed by a 3.

!! reexecutes the previous event

You can always reexecute the previous event by giving a !! command. In the following example, event 45 reexecutes event 44:

44 $ ls -l text
-rw-rw-r--   1 alex alex 45 Apr 30 14:53 text
45 $ !!
ls -l text
-rw-rw-r--   1 alex alex 45 Apr 30 14:53 text

The !! command works whether or not your prompt displays an event number. As this example shows, when you use the history mechanism to reexecute an event, the shell displays the command it is reexecuting.

!n event number

A number following an exclamation point refers to an event. If that event is in the history list, the shell executes it. Otherwise, the shell displays an error message. A negative number following an exclamation point references an event relative to the current event.For example, the command !3 refers to the third preceding event. After you issue a command, the relative event number of a given event changes (event 3 becomes event 4). Both of the following commands reexecute event 44:

51 $ !44
ls -l text
-rw-rw-r--  1 alex alex 45 Nov 30 14:53 text
52 $ !-8
ls -l text
-rw-rw-r--  1 alex alex 45 Nov 30 14:53 text

!string event text

When a string of text follows an exclamation point, the shell searches for and executes the most recent event that began with that string. If you enclose the string between question marks, the shell executes the most recent event that contained that string. The final question mark is optional if a RETURN would immediately follow it.

68 $ history 10
   59 ls -l text*
   60 tail text5
   61 cat text1 text5 > letter
   62 vim letter
   63 cat letter
   64 cat memo
   65 lpr memo
   66 pine jenny
   67 ls -l
   68 history
69 $ !l
ls -l
...
70 $ !lpr
lpr memo
71 $ !?letter?
cat letter
...

Optional: Word Designators

A word designator specifies a word or series of words from an event. Table 8-9 on page 302 lists word designators.

Table 8-9. Word designators

Designator

Meaning

n

The nth word. Word 0 is normally the command name.

^

The first word (after the command name).

$

The last word.

mn

All words from word number m through word number n; m defaults to 0 if you omit it (0n).

n*

All words from word number n through the last word.

*

All words except the command name. The same as 1*.

%

The word matched by the most recent ?string? search.

The words are numbered starting with 0 (the first word on the lineusually the command), continuing with 1 (the first word following the command), and going through n (the last word on the line).

To specify a particular word from a previous event, follow the event designator (such as !14) with a colon and the number of the word in the previous event. For example, !14:3 specifies the third word following the command from event 14. You can specify the first word following the command (word number 1) by using a caret (^) and the last word by using a dollar sign ($). You can specify a range of words by separating two word designators with a hyphen.

72 $ echo apple grape orange pear
apple grape orange pear
73 $ echo !72:2
echo grape
grape
74 $ echo !72:^
echo apple
apple
75 $ !72:0 !72:$
echo pear
pear
76 $ echo !72:2-4
echo grape orange pear
grape orange pear
77 $ !72:0-$
echo apple grape orange pear
apple grape orange pear

As the next example shows, !$ refers to the last word of the previous event. You can use this shorthand to edit, for example, a file you just displayed with cat:

$ cat report.718
...
$ vim !$
vim report.718
...

If an event contains a single command, the word numbers correspond to the argument numbers. If an event contains more than one command, this correspondence does not hold true for commands after the first. In the following example event 78 contains two commands separated by a semicolon so that the shell executes them sequentially; the semicolon is word number 5.

78 $ !72 ; echo helen jenny barbara
echo apple grape orange pear ; echo helen jenny barbara
apple grape orange pear
helen jenny barbara
79 $ echo !78:7
echo helen
helen
80 $ echo !78:4-7
echo pear ; echo helen
pear
helen

MODIFIERS

On occasion you may want to change an aspect of an event you are reexecuting. Perhaps you entered a complex command line with a typo or incorrect pathname or you want to specify a different argument. You can modify an event or a word of an event by putting one or more modifiers after the word designator, or after the event designator if there is no word designator. Each modifier must be preceded by a colon (:).

Substitute modifier

The substitute modifier is more complex than the other modifiers. The following example shows the substitute modifier correcting a typo in the previous event:

$ car /Users/jenny/memo.0507 /Users/alex/letter.0507
bash: car: command not found
$ !!:s/car/cat
cat /Users/jenny/memo.0507 /Users/alex/letter.0507
...

The substitute modifier has the following syntax:

[g]s/ old/new/

where old is the original string (not a regular expression), and new is the string that replaces old. The substitute modifier substitutes the first occurrence of old with new. Placing a g before the s (as in gs/old/new/) causes a global substitution, replacing all occurrences of old. The / is the delimiter in the examples but you can use any character that is not in either old or new. The final delimiter is optional if a RETURN would immediately follow it. As with the vim Substitute command, the history mechanism replaces an ampersand (&) in new with old. The shell replaces a null old string (s//new/) with the previous old string or string within a command that you searched for with ?string?.

Quick substitution

An abbreviated form of the substitute modifier is quick substitution. Use it to reexecute the most recent event while changing some of the event text. The quick substitution character is the caret (^). For example, the command

$ ^old^new^

produces the same results as

$ !!:s/old/new/

Thus substituting cat for car in the previous event could have been entered as

$ ^car^cat
cat /Users/jenny/memo.0507 /Users/alex/letter.0507
...

You can omit the final caret if it would be followed immediately by a RETURN. As with other command line substitutions, the shell displays the command line as it appears after the substitution.

Other modifiers

Modifiers (other than the substitute modifier) perform simple edits on the part of the event that has been selected by the event designator and the optional word designators. You can use multiple modifiers, each preceded by a colon (:).

The following series of commands uses ls to list the name of a file, repeats the command without executing it (p modifier), and repeats the last command, removing the last part of the pathname (h modifier) again without executing it:

$ ls /etc/postfix/aliases
/etc/postfix/aliases
$ !!:p
ls /etc/postfix/aliases
$ !!:h:p
ls /etc/postfix
$

Table 8-10 lists event modifiers other than the substitute modifier.

Table 8-10. Modifiers

Modifier

Function

e (extension)

Removes all but the filename extension

h (head)

Removes the last part of a pathname

p (print-not)

Displays the command, but does not execute it

q (quote)

Quotes the substitution to prevent further substitutions on it

r (root)

Removes the filename extension

t (tail)

Removes all elements of a pathname except the last

x

Like q but quotes each word in the substitution individually

The Readline Library

Command line editing under the Bourne Again Shell is implemented through the Readline Library, which is available to any application written in C. Any application that uses the Readline Library supports line editing that is consistent with that provided by bash. Programs that use the Readline Library, including bash, read ~/.inputrc (page 308) for key binding information and configuration settings. The noediting command line option turns off command line editing in bash.

vi mode

You can choose one of two editing modes when using the Readline Library in bash: emacs or vi(m). Both modes provide many of the commands available in the stand-alone versions of the vi(m) and emacs editors. You can also use the ARROW keys to move around. Up and down movements move you backward and forward through the history list. In addition, Readline provides several types of interactive word completion (page 306). The default mode is emacs; you can switch to vi mode with the following command:

$ set-o vi

emacs mode

The next command switches back to emacs mode:

$ set -o emacs

vi Editing Mode

Before you start make sure you are in vi mode.

When you enter bash commands while in vi editing mode, you are in Input mode (page 149). As you enter a command, if you discover an error before you press RETURN, you can press ESCAPE to switch to vi Command mode. This setup is different from the stand-alone vi(m) editor's initial mode. While in Command mode you can use many vi(m) commands to edit the command line. It is as though you were using vi(m) to edit a copy of the history file with a screen that has room for only one command. When you use the k command or the UP ARROW to move up a line, you access the previous command. If you then use the j command or the DOWN ARROW to move down a line, you will return to the original command. To use the k and j keys to move between commands you must be in Command mode; you can use the ARROW keys in both Command and Input modes.

Tip: The stand-alone editor starts in Command mode

The stand-alone vim editor starts in Command mode, whereas the command line vi (m) editor starts in Input mode. If commands display characters and do not work properly, you are in Input mode. Press ESCAPE and enter the command again.

In addition to cursor-positioning commands, you can use the search-backward (?) command followed by a search string to look back through your history list for the most recent command containing that string. If you have moved back in your history list, use a forward slash (/) to search forward toward your most recent command. Unlike the search strings in the stand-alone vi(m) editor, these search strings cannot contain regular expressions. You can, however, start the search string with a caret (^) to force the shell to locate commands that start with the search string. As in vi(m), pressing n after a successful search looks for the next occurrence of the same string.

You can also access events in the history list by using event numbers. While you are in Command mode (press ESCAPE), enter the event number followed by a G to go to the command with that event number.

When you use /, ?, or G to move to a command line, you are in Command mode, not Input mode. Now you can edit the command as you like or press RETURN to execute it.

Once the command you want to edit is displayed, you can modify the command line using vi(m) Command mode editing commands such as x (delete character), r (replace character), ~ (change case), and . (repeat last change). To change to Input mode, use an Insert (i, I), Append (a, A), Replace (R), or Change (c, C) command. You do not have to return to Command mode to run a command; simply press RETURN, even if the cursor is in the middle of the command line.

Refer to page 191 for a summary of vim commands.

emacs Editing Mode

Unlike the vi(m) editor, emacs is modeless. You need not switch between Command mode and Input mode because most emacs commands are control characters (page 208), allowing emacs to distinguish between input and commands. Like vi(m), the emacs command line editor provides commands for moving the cursor on the command line and through the command history list and for modifying part or all of a command. The emacs command line editor commands differ in a few cases from the commands in the stand-alone emacs editor.

In emacs you perform cursor movement by using both CONTROL and ESCAPE commands. To move the cursor one character backward on the command line, press CONTROL-B. Press CONTROL-F to move one character forward. As in vi, you may precede these movements with counts. To use a count you must first press ESCAPE; otherwise, the numbers you type will appear on the command line.

Like vi(m), emacs provides word and line movement commands. To move backward or forward one word on the command line, press ESCAPE b or ESCAPE f. To move several words by using a count, press ESCAPE followed by the number and the appropriate escape sequence. To get to the beginning of the line, press CONTROL-A; to the end of the line, press CONTROL-E; and to the next instance of the character c, press CONTROL-X CONTROL-F followed by c.

You can add text to the command line by moving the cursor to the correct place and typing the desired text. To delete text, move the cursor just to the right of the characters that you want to delete and press the erase key (page 25) once for each character you want to delete.

CONTROL-D can terminate your screen session

If you want to delete the character directly under the cursor, press CONTROL-D . If you enter CONTROL-D at the beginning of the line, it may terminate your shell session.

If you want to delete the entire command line, type the line kill character (page 25). You can type this character while the cursor is anywhere in the command line. If you want to delete from the cursor to the end of the line, use CONTROL-K.

Refer to page 243 for a summary of emacs commands.

Readline Completion Commands

You can use the TAB key to complete words you are entering on the command line. This facility, called completion, works in both vi and emacs editing modes and is similar to the completion facility available in tcsh. Several types of completion are possible, and which one you use depends on which part of a command line you are typing when you press TAB.

Command Completion

If you are typing the name of a command (the first word on the command line), pressing TAB results in command completion. That is, bash looks for a command whose name starts with the part of the word you have typed. If no command starts with what you have entered, bash beeps. If there is one such command, bash completes the command name for you. If there is more than one choice, bash does nothing in vi mode and beeps in emacs mode. Pressing TAB a second time causes bash to display a list of commands whose names start with the prefix you typed and allows you to finish typing the command name.

In the following example, the user types bz and presses TAB. The shell beeps (the user is in emacs mode) to indicate that several commands start with the letters bz. The user enters another TAB to cause the shell to display a list of commands that start with bz followed by the command line as the user had entered it so far:

$ bz    TAB (beep)    TAB
bzcat    bzdiff    bzip2     bzless
bzcmp     bzgrep    bzip2recover bzmore
$ bz

Next the user types c and presses TAB twice. The shell displays the two commands that start with bzc. The user types a followed by TAB and the shell then completes the command because only one command starts with bzca.

$ bz c    TAB (beep)    TAB
bzcat bzcmp
$ bzca    TAB    t

Pathname Completion

Pathname completion, which also uses TABs, allows you to type a portion of a pathname and have bash supply the rest. If the portion of the pathname that you have typed is sufficient to determine a unique pathname, bash displays that pathname. If more than one pathname would match it, bash completes the pathname up to the point where there are choices so that you can type more.

When you are entering a pathname, including a simple filename, and press TAB, the shell beeps (if the shell is in emacs modein vi mode there is no beep). It then extends the command line as far as it can.

$ cat films/dar_    TAB (beep) cat films/dark_

In the films directory every file that starts with dar has k_ as the next characters, so bash cannot extend the line further without making a choice among files. You are left with the cursor just past the _ character. At this point you can continue typing the pathname or press TAB twice. In the latter case bash beeps, displays your choices, redisplays the command line, and again leaves the cursor just after the _ character.

$ cat films/dark_    TAB (beep)    TAB
dark_passage dark_victory
$ cat films/dark_

When you add enough information to distinguish between the two possible files and press TAB, bash displays the unique pathname. If you enter p followed by TAB after the _ character, the shell completes the command line:

$ cat films/dark_ p    TAB    assage

Because there is no further ambiguity, the shell appends a SPACE so you can finish typing the command line or just press RETURN to execute the command. If the complete pathname is that of a directory, bash appends a slash (/) in place of a SPACE.

Variable Completion

When typing a variable name, pressing TAB results in variable completion, where bash tries to complete the name of the variable. In case of an ambiguity, pressing TAB twice displays a list of choices:

$ echo $HO    TAB    TAB
$HOME   $HOSTNAME $HOSTTYPE
$ echo $HO M    TAB    E

Caution: Pressing RETURN executes the command

Pressing RETURN causes the shell to execute the command regardless of where the cursor is on the command line.

.inputrc: Configuring Readline

The Bourne Again Shell and other programs that use the Readline Library read the file specified by the INPUTRC environment variable to obtain initialization information. If INPUTRC is not set, these programs read the ~/.inputrc file. They ignore lines of .inputrc that are blank or that start with a pound sign (#).

Variables

You can set variables in .inputrc to control the behavior of the Readline Library using the following syntax:

set variable value

Table 8-11 lists some variables and values you can use. See Readline Variables in the bash man or info page for a complete list.

Table 8-11. Readline variables

Variable

Effect

editing-mode

Set to vi to start Readline in vi mode. Set to emacs to start Readline in emacs mode (the default). Similar to the set o vi and set o emacs shell commands (page 304).

horizontal-scroll-mode

Set to on to cause long lines to extend off the right edge of the display area. Moving the cursor to the right when it is at the right edge of the display area shifts the line to the left so you can see more of the line. You can shift the line back by moving the cursor back past the left edge. The default value is off, which causes long lines to wrap onto multiple lines of the display.

mark-directories

Set to off to cause Readline not to place a slash (/) at the end of directory names it completes. Normally it is on.

mark-modified-lines

Set to on to cause Readline to precede modified history lines with an asterisk. The default value is off.

Key Bindings

You can specify bindings that map keystroke sequences to Readline commands, allowing you to change or extend the default bindings. As in emacs, the Readline Library includes many commands that are not bound to a keystroke sequence. To use an unbound command, you must map it using one of the following forms:

keyname: command_name
"keystroke_sequence" : command_name

In the first form, you spell out the name for a single key. For example, CONTROL-U would be written as control-u. This form is useful for binding commands to single keys.

In the second form, you specify a string that describes a sequence of keys that will be bound to the command. You can use the emacs-style backslash escape sequences to represent the special keys CONTROL (\C), META (\M), and ESCAPE (\e). Specify a backslash by escaping it with another backslash: \\. Similarly, a double or single quotation mark can be escaped with a backslash: \ "or \'.

The kill-whole-line command, available in emacs mode only, deletes the current line. Put the following command in .inputrc to bind the kill-whole-line command (which is unbound by default) to the keystroke sequence CONTROL-R.

control-r: kill-whole-line

bind

Give the command bind P to display a list of all Readline commands. If a command is bound to a key sequence, that sequence is shown. Commands you can use in vi mode start with vi. For example, vi-next-word and vi-prev-word move the cursor to the beginning of the next and previous words, respectively. Commands that do not begin with vi are generally available in emacs mode.

Use bind q to determine which key sequence is bound to a command:

$ bind -q kill-whole-line
kill-whole-line can be invoked via "\C-r".

You can also bind text by enclosing it within double quotation marks (emacs mode only):

"QQ": "The Mac OS X Operating System"

This command causes bash to insert the string The Mac OS X Operating System when you type QQ.

Conditional Constructs

You can conditionally select parts of the .inputrc file using the $if directive. The syntax of the conditional construct is

$if test[= value]
           commands
    [$else
           commands]
$endif

where test is mode, term, or bash. If test equals value or if test is true, this structure executes the first set of commands. If test does not equal value or if test is false, it executes the second set of commands if they are present or exits from the structure if they are not present.

The power of the $if directive lies in the three types of tests it can perform.

  1. You can test to see which mode is currently set.

    $ if mode=vi

    The preceding test is true if the current Readline mode is vi and false otherwise. You can test for vi or emacs.

  2. You can test the type of terminal.

    $ if term=xterm

    The preceding test is true if the TERM variable is set to xterm. You can test for any value of TERM.

  3. You can test the application name.

    $ if bash

    The preceding test is true when you are running bash and not another program that uses the Readline Library. You can test for any application name.

These tests can customize the Readline Library based on the current mode, the type of terminal, and the application you are using. They give you a great deal of power and flexibility when using the Readline Library with bash and other programs.

The following commands in .inputrc cause CONTROL-Y to move the cursor to the beginning of the next word regardless of whether bash is in vi or emacs mode:

$ cat ~/.inputrc
set editing-mode vi
$if mode=vi
       "\C-y": vi-next-word
    $else
       "\C-y": forward-word
$endif

Because bash reads the preceding conditional construct when it is started, you must set the editing mode in .inputrc. Changing modes interactively using set will not change the binding of CONTROL-Y.

For more information on the Readline Library, open the bash man page and give the command /^READLINE, which searches for the word READLINE at the beginning of a line.

Tip: If Readline commands do not work, log out and log in again

The Bourne Again Shell reads ~/.inputrc when you log in. After you make changes to this file, you should log out and log in again before testing the changes.


Aliases

An alias is a (usually short) name that the shell translates into another (usually longer) name or (complex) command. Aliases allow you to define new commands by substituting a string for the first token of a simple command. They are typically placed in the ~/.bashrc (bash) or ~/.tcshrc (tcsh) startup files so that they are available to interactive subshells.

Under bash the syntax of the alias builtin is

alias [name[= value]]

Under tcsh the syntax is

alias [name[ value]]

In the bash syntax there are no SPACEs around the equal sign. If value contains SPACEs or TABs, you must enclose value between quotation marks. Unlike aliases under tcsh, a bash alias does not accept an argument from the command line in value. Use a bash function (page 314) when you need to use an argument.

An alias does not replace itself, which avoids the possibility of infinite recursion in handling an alias such as the following:

$ alias ls='ls -F'

You can nest aliases. Aliases are disabled for noninteractive shells (that is, shell scripts). To see a list of the current aliases, give the command alias. To view the alias for a particular name, use alias followed by the name and nothing else. You can use the unalias builtin to remove an alias.

When you give an alias builtin without any arguments, the shell displays a list of all defined aliases:

$ alias
alias ll='ls -l'
alias l='ls -ltr'
alias ls='ls -F'
alias zap='rm -i'

Give an alias command to see which aliases are in effect. You can delete the aliases you do not want from the appropriate startup file.

Single Versus Double Quotation Marks in Aliases

The choice of single or double quotation marks is significant in the alias syntax when the alias includes variables. If you enclose value within double quotation marks, any variables that appear in value are expanded when the alias is created. If you enclose value within single quotation marks, variables are not expanded until the alias is used. The following example illustrates the difference.

The PWD keyword variable holds the pathname of the working directory. Alex creates two aliases while he is working in his home directory. Because he uses double quotation marks when he creates the dirA alias, the shell substitutes the value of the working directory when he creates this alias. The alias dirA command displays the dirA alias and shows that the substitution has already taken place:

$ echo $PWD
/Users/alex
$ alias dirA="echo Working directory is $PWD"
$ alias dirA
alias dirA='echo Working directory is /Users/alex'

When Alex creates the dirB alias, he uses single quotation marks, which prevent the shell from expanding the $PWD variable. The alias dirB command shows that the dirB alias still holds the unexpanded $PWD variable:

$ alias dirB='echo Working directory is $PWD'
$ alias dirB
alias dirB='echo Working directory is $PWD'

After creating the dirA and dirB aliases, Alex uses cd to make cars his working directory and gives each of the aliases as commands. The alias that he created with double quotation marks displays the name of the directory that he created the alias in as the working directory (which is wrong) and the dirB alias displays the proper name of the working directory:

$ cd cars
$ dirA
Working directory is /Users/alex
$ dirB
Working directory is /Users/alex/cars

Tip: How to prevent the shell from invoking an alias

The shell checks only simple, unquoted commands to see if they are aliases. Commands given as relative or absolute pathnames and quoted commands are not checked. When you want to give a command that has an alias but do not want to use the alias, precede the command with a backslash, specify the command's absolute pathname, or give the command as ./command.

Examples of Aliases

The following alias allows you to type r to repeat the previous command or r abc to repeat the last command line that began with abc:

$ alias r='fc -s'

If you use the command ls 1tr frequently, you can create an alias that substitutes ls 1tr when you give the command l:

$ alias l='ls -ltr'
$ l
total 41
-rw-r--r-- 1 alex  alex  30015  Mar  1 2004 flute.ps
-rw-r----- 1 alex  alex   3089  Feb 11 2005 XTerm.ad
-rw-r--r-- 1 alex  alex    641  Apr  1 2005 fixtax.icn
-rw-r--r-- 1 alex  alex   484 Apr  9 2005 maptax.icn
drwxrwxr-x 2 alex  alex  1024 Aug  9 17:41 Tiger
drwxrwxr-x 2 alex  alex  1024 Sep 10 11:32 testdir
-rwxr-xr-x 1 alex  alex   485 Oct 21 08:03 floor
drwxrwxr-x 2 alex  alex  1024 Oct 27 20:19 Test_Emacs

Another common use of aliases is to protect yourself from mistakes. The following example substitutes the interactive version of the rm utility when you give the command zap:

$ alias zap='rm -i'
$ zap f*
rm: remove 'fixtax.icn'? n
rm: remove 'flute.ps'? n
rm: remove 'floor'? n

The i option causes rm to ask you to verify each file that would be deleted, to help you avoid accidentally deleting the wrong file. You can also alias rm with the rmi command: alias rm='rm i'.

The aliases in the next example cause the shell to substitute ls 1 each time you give an ll command and ls F when you use ls:

$ alias ls='ls -F'
$ alias ll='ls -l'
$ ll
total 41
drwxrwxr-x 2 alex  alex   1024 Oct 27 20:19 Test_Emacs/
drwxrwxr-x 2 alex  alex   1024 Aug 9 17:41 Tiger/
-rw-r----- 1 alex  alex   3089 Feb 11 2005 XTerm.ad
-rw-r--r-- 1 alex  alex    641 Apr 1 2005 fixtax.icn
-rw-r--r-- 1 alex  alex   30015 Mar 1 2004 flute.ps
-rwxr-xr-x 1 alex  alex    485 Oct 21 08:03 floor*
-rw-r--r-- 1 alex  alex    484 Apr 9 2005 maptax.icn
drwxrwxr-x 2 alex  alex   1024 Sep 10 11:32 testdir/

The F option causes ls to print a slash (/) at the end of directory names and an asterisk (*) at the end of the names of executable files. In this example, the string that replaces the alias ll (ls 1) itself contains an alias (ls). When it replaces an alias with its value, the shell looks at the first word of the replacement string to see whether it is an alias. In the preceding example, the replacement string contains the alias ls, so a second substitution occurs to produce the final command ls F 1. (To avoid a recursive plunge, the ls in the replacement text, although an alias, is not expanded a second time.)

When given a list of aliases without the =value or value field, the alias builtin responds by displaying the value of each defined alias. The alias builtin reports an error if an alias has not been defined:

$ alias ll l ls zap wx
alias ll='ls -l'
alias l='ls -ltr'
alias ls='ls -F'
alias zap='rm -i'
bash: alias: wx: not found

You can avoid alias substitution by preceding the aliased command with a backslash (\):

$ \ls
Test_Emacs XTerm.ad flute.ps maptax.icn
Tiger  fixtax.icn floor testdir

Because the replacement of an alias name with the alias value does not change the rest of the command line, any arguments are still received by the command that gets executed:

$ ll f*
-rw-r--r-- 1 alex alex    641 Apr  1 2005 fixtax.icn
-rw-r--r-- 1 alex alex  30015 Mar 1 2004 flute.ps
-rwxr-xr-x 1 alex  alex   485 Oct 21 08:03 floor*

You can remove an alias with the unalias builtin. When the zap alias is removed, it is no longer displayed with the alias builtin and its subsequent use results in an error message:

$ unalias zap
$ alias
alias ll='ls -l'
alias l='ls -ltr'
alias ls='ls -F'
$ zap maptax.icn
bash: zap: command not found


Functions

A shell function (tcsh does not have functions) is similar to a shell script in that it stores a series of commands for execution at a later time. However, because the shell stores a function in the computer's main memory (RAM) instead of in a file on the disk, the shell can access it more quickly than the shell can access a script. The shell also preprocesses (parses) a function so that it starts up more quickly than a script. Finally the shell executes a shell function in the same shell that called it. If you define too many functions, the overhead of starting a subshell (as when you run a script) can become unacceptable.

You can declare a shell function in the ~/.bash_profile startup file, in the script that uses it, or directly from the command line. You can remove functions with the unset builtin. The shell does not keep functions once you log out.

Tip: Removing variables and functions

If you have a shell variable and a function with the same name, using unset removes the shell variable. If you then use unset again with the same name, it removes the function.

The syntax that declares a shell function is

[function] function-name ()
{
commands
}

where the word function is optional, function-name is the name you use to call the function, and commands comprise the list of commands the function executes when you call it. The commands can be anything you would include in a shell script, including calls to other functions.

The first brace ({) can appear on the same line as the function name. Aliases and variables are expanded when a function is read, not when it is executed. You can use the break statement (page 543) within a function to terminate its execution.

Shell functions are useful as a shorthand as well as to define special commands. The following function starts a process named process in the background, with the output normally displayed by process being saved in .process.out:

start_process( ) {
process > .process.out 2>&1 &
}

The next example shows how to create a simple function that displays the date, a header, and a list of the people who are using the system. This function runs the same commands as the whoson script described on page 265. In this example the function is being entered from the keyboard. The greater-than (>) signs are secondary shell prompts (PS2); do not enter them.

$ function whoson ()
> {
>  date
>  echo "Users Currently Logged On"
>  who
> }

$ whoson
Sun Aug 7 15:44:58 PDT 2005
Users Currently Logged On
zach     console  May 5 22:18
zach     ttyp1    May 5 22:20
zach     ttyp2    May 5 22:21 (bravo.example.co)

Functions in startup files

If you want to have the whoson function always be available without having to enter it each time you log in, put its definition in ~/.bash_profile. Then run .bash_profile, using the . (dot) command to put the changes into effect immediately:

$ cat ~/.bash_profile
export
TERM=vt100
stty kill '^u'
whoson ()
{
    date
    echo "Users Currently Logged On"
    who
}
$ . ~/.bash_profile

You can specify arguments when you call a function. Within the function these arguments are available as positional parameters (page 564). The following example shows the arg1 function entered from the keyboard.

$ arg1 ( ) {
> echo "$1"
> }
$ arg1 first_arg
first_arg

See the function switch () on page 261 for another example of a function. "Functions" on page 561 discusses the use of local and global variables within a function.

Optional

The following function allows you to export variables using tcsh syntax. The env builtin lists all environment variables and their values and verifies that setenv worked correctly:

$ cat .bash_profile
...
# setenv - keep tcsh users happy
function setenv( )
{
    if [ $# -eq 2 ]
       then
          eval $1=$2
          export $1
       else
          echo "Usage: setenv NAME VALUE" 1>&2
    fi
}
     $ . ~/.bash_profile
     $ setenv TCL_LIBRARY /usr/local/lib/tcl
     $ env | grep TCL_LIBRARY
     TCL_LIBRARY=/usr/local/lib/tcl

eval

The $# special parameter (page 564) takes on the value of the number of command line arguments. This function uses the eval builtin to force bash to scan the command $1=$2 twice. Because $1=$2 begins with a dollar sign ($), the shell treats the entire string as a single tokena command. With variable substitution performed, the command name becomes TCL_LIBRARY=/usr/local/lib/tcl, which results in an error. Using eval, a second scanning splits the string into the three desired tokens, and the correct assignment occurs. See page 374 for more information on eval.


Controlling bash Features and Options

This section explains how to control bash features and options using command line options and the set and shopt builtins.

Command Line Options

Two kinds of command line options are available: short and long. Short options consist of a hyphen followed by a letter; long options have two hyphens followed by multiple characters. Long options must appear before short options on a command line that calls bash. Table 8-12 lists some commonly used command line options.

Table 8-12. Command line options

Option

Explanation

Syntax

Help

Displays a usage message.

--help

No edit

Prevents users from using the Readline Library (page 304) to edit command lines in an interactive shell.

--noediting

No profile

Prevents reading these startup files (page 259): /etc/profile, ~/.bash_profile, ~/.bash_login, and ~/.profile.

--noprofile

No rc

Prevents reading the ~/.bashrc startup file (page 260). This option is on by default if the shell is called as sh.

--norc

POSIX

Runs bash in POSIX mode.

--posix

Version

Displays bash version information and exits.

--version

Login

Causes bash to run as though it were a login shell.

--I (lowercase "I")

shopt

Runs a shell with the opt shopt option (page 318). A -0 (uppercase "0") sets the option; +0 unsets it.

[±]0 [opt]

End of options

On the command line, signals the end of options. Subsequent tokens are treated as arguments even if they begin with a hyphen (-).

Shell Features

You can control the behavior of the Bourne Again Shell by turning features on and off. Different features use different methods to turn features on and off. The set builtin controls one group of features, while the shopt builtin controls another group. You can also control many features from the command line you use to call bash.

Tip: Features, options, variables?

To avoid confusing terminology, this book refers to the various shell behaviors that you can control as features. The bash info page refers to them as "options" and "values of variables controlling optional shell behavior."

set ±o: Turns Shell Features On and Off

The set builtin (there is a set builtin in tcsh, but it works differently), when used with the -o or +o option, enables, disables, and lists certain bash features. For example, the following command turns on the noclobber feature (page 125):

$ set -o noclobber

You can turn this feature off (the default) by giving the command

$ set +o noclobber

The command set -o without an option lists each of the features controlled by set followed by its state (on or off). The command set +o without an option lists the same features in a form that you can use as input to the shell. Table 8-13 lists bash features.

Table 8-13. bash features

Feature

Description

Syntax

Alternate syntax

allexport

Automatically exports all variables and functions that you create or modify after giving this command.

set o allexport

set a

braceexpand

Causes bash to perform brace expansion (the default; page 323).

set o braceexpand

set B

cdspell

Corrects minor spelling errors in directory names used as arguments to cd.

shopt s cdspell

cmdhist

Saves all lines of a multiline command in the same history entry, adding semicolons as needed.

shopt s cmdhist

dotglob

Causes shell special characters (wildcards; page 133) in an ambiguous file reference to match a leading period in a filename. By default special characters do not to match a leading period. You must always specify the filenames . and .. explicitly because no pattern ever matches them.

shopt s dotglob

emacs

Specifies emacs editing mode for command line editing (the default; page 305).

set o emacs

errexit

Causes bash to exit when a simple command (not a control structure) fails.

set o errexit

set e

execfail

Causes a shell script to continue running when it cannot find the file that is given as an argument to exec. By default a script terminates when exec cannot find the file that is given as its argument.

shopt s execfail

expand_aliases

Causes aliases (page 311) to be expanded (by default it is on for interactive shells and off for noninteractive shells).

shopt s expand_alias

hashall

Causes bash to remember where commands it has found using PATH (page 285) are located (default).

set o hashall

set h

histappend

Causes bash to append the history list to the file named by HISTFILE (page 295) when the shell exits. By default bash overwrites this file.

shopt s histappend

histexpand

Causes the history mechanism (which uses exclamation points; page 299) to work (default). Turn this feature off to turn off history expansion.

set o histexpand

set H

history

Enable command history (on by default; page 295).

set o history

ignoreeof

Specifies that bash must receive ten EOF characters before it exits. Useful on noisy dial-up lines.

set o ignoreeof

monitor

Enables job control (on by default, page 272).

set o monitor

set m

nocaseglob

Causes ambiguous file references (page 133) to match filenames without regard to case (off by default).

shopt s nocaseglob

noclobber

Helps prevent overwriting files (off by default; page 125).

set o noclobber

set C

noglob

Disables pathname expansion (off by default; page 133).

set o noglob

set f

notify

With job control (page 272) enabled, reports the termination status of background jobs immediately. The default behavior is to display the status just before the next prompt.

set o notify

set b

nounset

Displays an error and exits from a shell script when you use an unset variable in an interactive shell. The default is to display a null value for an unset variable.

set o nounset

set u

nullglob

Causes bash to expand ambiguous file references (page 133) that do not match a filename to a null string. By default bash passes these file references without expanding them.

shopt s nullglob

posix

Runs bash in POSIX mode.

set o posix

verbose

Displays command lines as bash reads them.

set o verbose

set v

vi

Specifies vi editing mode for command line editing (page 304).

set o vi

xpg_echo

Causes the echo builtin to expand back-slash escape sequences without the need for the e option (page 548).

shopt s xpg_echo

xtrace

Turns on shell debugging (page 536).

set o xtrace

set x

shopt: Turns Shell Features On and Off

The shopt (shell option) builtin (not available in tcsh) enables, disables, and lists certain bash features that control the behavior of the shell. For example, the following command causes bash to include filenames that begin with a period (.) when it expands ambiguous file references (the -s stands for set):

$ shopt -s dotglob

You can turn this feature off (the default) by giving the command (the u stands for unset)

$ shopt -u dotglob

The shell displays how a feature is set if you give the name of the feature as the only argument to shopt:

$ shopt dotglob
dotglob           off

The command shopt without any options or arguments lists the features controlled by shopt and their state. The command shopt -s without an argument lists the features controlled by shopt that are set or on. The command shopt u lists the features that are unset or off. Table 8-13 lists bash features.

Tip: Setting set ±0 features using shopt

You can use shopt to set/unset features that are otherwise controlled by set ±o. Use the regular shopt syntax with s or u and include the o option. For example, the following command turns on the noclobber feature:

$ shopt -o -s noclobber


Processing the Command Line

Whether you are working interactively or running a shell script, bash needs to read a command line before it can start processing itbash always reads at least one line before processing a command. Some bash builtins, such as if and case, as well as functions and quoted strings, span multiple lines. When bash recognizes a command that covers more than one line, it reads the entire command before processing it. In interactive sessions bash prompts you with the secondary prompt (PS2, > by default; page 288) as you type each line of a multiline command until it recognizes the end of the command:

$ echo 'hi
> end'
hi
end
$ function hello () {
> echo hello there
> }
$

After reading a command line, bash applies history expansion and alias substitution to the line.

History Expansion

"Reexecuting and Editing Commands" on page 296 discusses the commands you can give to modify and reexecute command lines from the history list. History expansion is the process that bash uses to turn a history command into an executable command line. For example, when you give the command !!, history expansion changes that command line so it is the same as the previous one. History expansion is turned on by default for interactive shells; set +o histexpand turns it off. History expansion does not apply to noninteractive shells (shell scripts).

Alias Substitution

Aliases (page 311) substitute a string for the first word of a simple command. By default aliases are turned on for interactive shells and off for noninteractive shells. Give the command shopt u expand_aliases to turn aliases off.

Parsing and Scanning the Command Line

After processing history commands and aliases, bash does not execute the command immediately. One of the first things the shell does is to parse (isolate strings of characters in) the command line into tokens or words. The shell then scans each token for special characters and patterns that instruct the shell to take certain actions. These actions can involve substituting one word or words for another. When the shell parses the following command line, it breaks it into three tokens (cp, ~/letter, and .):

$ cp ~/letter .

After separating tokens and before executing the command, the shell scans the tokens and performs command line expansion.

Command Line Expansion

In both interactive and noninteractive use, the shell transforms the command line using command line expansion before passing the command line to the program being called. You can use a shell without knowing much about command line expansion, but you can use what a shell has to offer to a better advantage with an understanding of this topic. This section covers Bourne Again Shell command line expansion; TC Shell command line expansion is covered starting on page 341.

The Bourne Again Shell scans each token for the various types of expansion and substitution in the following order. Most of these processes expand a word into a single word. Only brace expansion, word splitting, and pathname expansion can change the number of words in a command (except for the expansion of the variable "$@"page 566).

  1. Brace expansion (page 323)

  2. Tilde expansion (page 324)

  3. Parameter and variable expansion (page 325)

  4. Arithmetic expansion (page 325)

  5. Command substitution (page 327)

  6. Word splitting (page 328)

  7. Pathname expansion (page 329)

  8. Process substitution (page 330)

Quote removal

After bash finishes with the preceding list, it removes from the command line single quotation marks, double quotation marks, and backslashes that are not a result of an expansion. This process is called quote removal.

Order of expansion

The order in which bash carries out these steps affects the interpretation of commands. For example, if you set a variable to a value that looks like the instruction for output redirection and then enter a command that uses the variable's value to perform redirection, you might expect bash to redirect the output.

$ SENDIT="> /tmp/saveit"
$ echo xxx $SENDIT
xxx > /tmp/saveit
$ cat /tmp/saveit
cat: /tmp/saveit: No such file or directory

In fact, the shell does not redirect the outputit recognizes input and output redirection before it evaluates variables. When it executes the command line, the shell checks for redirection and, finding none, evaluates the SENDIT variable. After replacing the variable with > /tmp/saveit, bash passes the arguments to echo, which dutifully copies its arguments to standard output. No /tmp/saveit file is created.

The following sections provide more detailed descriptions of the steps involved in command processing. Keep in mind that double and single quotation marks cause the shell to behave differently when performing expansions. Double quotation marks permit parameter and variable expansion but suppress other types of expansion. Single quotation marks suppress all types of expansion.

Brace expansion

Brace expansion, which originated in the C Shell, provides a convenient way to specify filenames when pathname expansion does not apply. Although brace expansion is almost always used to specify filenames, the mechanism can be used to generate arbitrary strings; the shell does not attempt to match the brace notation with the names of existing files.

Brace expansion is turned on in interactive and noninteractive shells by default; you can turn it off with set +o braceexpand. The shell also uses braces to isolate variable names (page 281).

The following example illustrates how brace expansion works. The ls command does not display any output because there are no files in the working directory. The echo builtin displays the strings that the shell generates with brace expansion. In this case the strings do not match filenames (there are no files in the working directory.)

$ ls
$ echo chap_{one,two,three}.txt
chap_one.txt chap_two.txt chap_three.txt

The shell expands the comma-separated strings inside the braces in the echo command into a SPACE-separated list of strings. Each string from the list is prepended with the string chap_, called the preamble, and appended with the string .txt, called the postscript. Both the preamble and the postscript are optional. The left-to-right order of the strings within the braces is preserved in the expansion. For the shell to treat the left and right braces specially and for brace expansion to occur, at least one comma and no unquoted whitespace characters must be inside the braces. You can nest brace expansions.

Brace expansion is useful when there is a long preamble or postscript. The following example copies the four files main.c, f1.c, f2.c, and tmp.c located in the /usr/local/src/C directory to the working directory:

$ cp /usr/local/src/C/{main,f1,f2,tmp}.c .

You can also use brace expansion to create directories with related names:

$ ls -F
file1  file2  file3
$ mkdir vrs{A,B,C,D,E}
$ ls -F
file1  file2  file3 vrsA/ vrsB/ vrsC/ vrsD/ vrsE/

The F option causes ls to display a slash (/) after a directory and an asterisk (*) after an executable file.

If you tried to use an ambiguous file reference instead of braces to specify the directories, the result would be different (and not what you wanted):

$ rmdir vrs*
$ mkdir vrs[A-E]
$ ls -F
file1  file2  file3  vrs[A-E]/

An ambiguous file reference matches the names of existing files. Because it found no filenames matching vrs[AE], bash passed the ambiguous file reference to mkdir, which created a directory with that name. Page 136 has a discussion of brackets in ambiguous file references.

Tilde expansion

Chapter 4 (page 78) showed a shorthand notation to specify your home directory or the home directory of another user. This section provides a more detailed explanation of tilde expansion.

The tilde (~) is a special character when it appears at the start of a token on a command line. When it sees a tilde in this position, bash looks at the following string of charactersup to the first slash (/) or to the end of the word if there is no slashas a possible login name. If this possible login name is null (that is, if the tilde appears as a word by itself or if it is immediately followed by a slash), the shell substitutes the value of the HOME variable for the tilde. The following example demonstrates this expansion, where the last command copies the file named letter from Alex's home directory to the working directory:

$ echo $HOME
/Users/alex
$ echo ~
/Users/alex
$ echo ~/letter
/Users/alex/letter
$ cp ~/letter .

If the string of characters following the tilde forms a valid login name, the shell substitutes the path of the home directory associated with that login name for the tilde and name. If it is not null and not a valid login name, the shell does not make any substitution:

$ echo ~jenny
/Users/jenny
$ echo ~root
/var/root
$ echo ~xx
~xx

Tildes are also used in directory stack manipulation (page 275). In addition, ~+ is a synonym for PWD (the name of the working directory), and ~ is a synonym for OLDPWD (the name of the previous working directory).

Parameter and Variable Expansion

On a command line a dollar sign ($) that is not followed by an open parenthesis introduces parameter or variable expansion. Parameters include command line, or positional, parameters (page 564) and special parameters (page 562). Variables include user-created variables (page 279) and keyword variables (page 284). The bash man and info pages do not make this distinction, however.

Parameters and variables are not expanded if they are enclosed within single quotation marks or if the leading dollar sign is escaped (preceded with a backslash). If they are enclosed within double quotation marks, the shell expands parameters and variables.

Arithmetic Expansion

The shell performs arithmetic expansion by evaluating an arithmetic expression and replacing it with the result. See page 355 for information on arithmetic expansion under tcsh. Under bash the syntax for arithmetic expansion is

$ ((expression))

The shell evaluates expression and replaces $((expression)) with the result of the evaluation. This syntax is similar to the syntax used for command substitution [$(...)] and performs a parallel function. You can use $((expression)) as an argument to a command or in place of any numeric value on a command line.

The rules for forming expression are the same as those found in the C programming language; all standard C arithmetic operators are available (see Table 13-8 on page 588). Arithmetic in bash is done using integers. Unless you use variables of type integer (page 284) or actual integers, however, the shell must convert string-valued variables to integers for the purpose of the arithmetic evaluation.

You do not need to precede variable names within expression with a dollar sign ($). In the following example, an arithmetic expression determines how many years are left until age 60:

$ cat age_check
#!/bin/bash
echo -n "How old are you? "
read age
echo "Wow, in $((60-age)) years, you'll be 60!"

$ age_check
How old are you? 55
Wow, in 5 years, you'll be 60!

You do not need to enclose the expression within quotation marks because bash does not perform filename expansion on it. This feature makes it easier for you to use an asterisk (*) for multiplication, as the following example shows:

$ echo There are $((60*60*24*365)) seconds in a non-leap year.
There are 31536000 seconds in a non-leap year.

Tip: Fewer dollar signs ($)

When you use variables within $(( and )), the dollar signs that precede individual variable references are optional:

$ x=23 y=37
$ echo $((2*$x + 3*$y))
157
$ echo $((2*x + 3*y))
157

The next example uses wc, cut, arithmetic expansion, and command substitution to estimate the number of pages required to print the contents of the file letter.txt. The output of the wc utility (page 888) used with the l option is the number of lines in the file, in columns 1 through 4, followed by a SPACE and the name of the file (the first command following). The cut utility (page 699) with the c1-4 option extracts the first four columns.

$ wc -l letter.txt
351 letter.txt
$ wc -l letter.txt | cut -c1-4
351

The dollar sign and single parenthesis instruct the shell to perform command substitution; the dollar sign and double parentheses indicate arithmetic expansion:

$ echo $(( $(wc -l letter.txt | cut -c1-4)/66 + 1))
6

The preceding example sends standard output from wc to standard input of cut via a pipe. Because of command substitution, the output of both commands replaces the commands between the $( and the matching ) on the command line. Arithmetic expansion then divides this number by 66, the number of lines on a page. A 1 is added at the end because the integer division results in any remainder being discarded.

Another way to get the same result without using cut is to redirect the input to wc instead of having wc get its input from a file you name on the command line. When you redirect its input, wc does not display the name of the file:

$ wc -l < letter.txt
       351

It is common practice to assign the result of arithmetic expansion to a variable:

$ numpages=$(($(wc -l < letter.txt)/66 + 1))

let builtin

The let builtin (not available in tcsh) evaluates arithmetic expressions just as the $(( )) syntax does. The following command is equivalent to the preceding one:

$ let "numpages=$(wc -l < letter.txt)/66 + 1"

The double quotation marks keep the SPACEs (both those you can see and those that result from the command substitution) from separating the expression into separate arguments to let. The value of the last expression determines the exit status of let. If the value of the last expression is 0, the exit status of let is 1; otherwise, the exit status is 0.

You can give multiple arguments to let on a single command line:

$ let a=5+3 b=7+2
$ echo $a $b
8 9

When you refer to variables when doing arithmetic expansion with let or $(()), the shell does not require you to begin the variable name with a dollar sign ($). Nevertheless, it is a good practice to do so, as in most places you must include this symbol.

Command Substitution

Command substitution replaces a command with the output of that command. The preferred syntax for command substitution under bash follows:

$(command)

Under bash you can also use the following syntax, which is the only syntax allowed under tcsh:

'command'

The shell executes command within a subshell and replaces command, along with the surrounding punctuation, with standard output of command.

In the following example, the shell executes pwd and substitutes the output of the command for the command and surrounding punctuation. Then the shell passes the output of the command, which is now an argument, to echo, which displays it.

$ echo $(pwd)
/Users/alex

The next script assigns the output of the pwd builtin to the variable where and displays a message containing the value of this variable:

$ cat where
where=$(pwd)
echo "You are using the $where directory."
$ where
You are using the /Users/jenny directory.

Although it illustrates how to assign the output of a command to a variable, this example is not realistic. You can more directly display the output of pwd without using a variable:

$ cat where2
echo "You are using the $(pwd) directory."
$ where2
You are using the /Users/jenny directory.

The following command uses find to locate files with the name README in the directory tree with its root at the working directory. This list of files is standard output of find and becomes the list of arguments to ls.

$ ls -l $(find . -name README -print)

The next command line shows the older 'command' syntax:

$ ls -l 'find . -name README -print'

One advantage of the newer syntax is that it avoids the rather arcane rules for token handling, quotation mark handling, and escaped back ticks within the old syntax. Another advantage of the new syntax is that it can be nested, unlike the old syntax. For example, you can produce a long listing of all README files whose size exceeds the size of ./README with the following command:

$ ls -l $(find . -name README -size +$(echo $(cat ./README | wc -c)c ) -print )

Try giving this command after giving a set x command (page 536) to see how bash expands it. If there is no README file, you just get the output of ls l.

For additional scripts that use command substitution, see pages 532, 549, and 579.

Tip: $((Versus $(

The symbols $(( constitute a separate token. They introduce an arithmetic expression, not a command substitution. Thus, if you want to use a parenthesized subshell (page 271) within $(), you must insert a SPACE between the $( and the next (.

Word Splitting

The results of parameter and variable expansion, command substitution, and arithmetic expansion are candidates for word splitting. Using each character of IFS (page 288) as a possible delimiter, bash splits these candidates into words or tokens. If IFS is unset, bash uses its default value (SPACE-TAB-NEWLINE). If IFS is null, bash does not split words.

Pathname Expansion

Pathname expansion (page 133), also called filename generation or globbing, is the process of interpreting ambiguous file references and substituting the appropriate list of filenames. Unless noglob (page 320) is set, the shell performs this function when it encounters an ambiguous file referencea token containing any of the unquoted characters *,?, [, or ]. If bash cannot locate any files that match the specified pattern, the token with the ambiguous file reference is left alone. The shell does not delete the token or replace it with a null string but rather passes it to the program as is (except see nullglob on page 320). The TC Shell generates an error message.

In the first echo command in the following example, the shell expands the ambiguous file reference tmp* and passes three tokens (tmp1, tmp2, and tmp3) to echo. The echo builtin displays the three filenames it was passed by the shell. After rm removes the three tmp* files, the shell finds no filenames that match tmp* when it tries to expand it. Thus it passes the unexpanded string to the echo builtin, which displays the string it was passed.

$ ls
tmp1 tmp2 tmp3
$ echo tmp*
tmp1 tmp2 tmp3
$ rm tmp*
$ echo tmp*
tmp*

By default the same command causes the TC Shell to display an error message:

tcsh $ echo tmp*
echo: No match

A period that either starts a pathname or follows a slash (/) in a pathname must be matched explicitly unless you have set dotglob (page 319). The option nocaseglob (page 320) causes ambiguous file references to match filenames without regard to case.

Quotation marks

Putting double quotation marks around an argument causes the shell to suppress pathname and all other expansion except parameter and variable expansion. Putting single quotation marks around an argument suppresses all types of expansion. The second echo command in the following example shows the variable $alex between double quotation marks, which allow variable expansion. As a result the shell expands the variable to its value: sonar. This expansion does not occur in the third echo command, which uses single quotation marks. Because neither single nor double quotation marks allow pathname expansion, the last two commands display the unexpanded argument tmp*.

$ echo tmp* $alex
tmp1 tmp2 tmp3 sonar
$ echo "tmp* $alex"
tmp* sonar
$ echo 'tmp* $alex'
tmp* $alex

The shell distinguishes between the value of a variable and a reference to the variable and does not expand ambiguous file references if they occur in the value of a variable. As a consequence you can assign to a variable a value that includes special characters, such as an asterisk (*).

Levels of expansion

In the next example, the working directory has three files whose names begin with letter. When you assign the value letter* to the variable var, the shell does not expand the ambiguous file reference because it occurs in the value of a variable (in the assignment statement for the variable). No quotation marks surround the string letter*; context alone prevents the expansion. After the assignment the set builtin (with the help of grep) shows the value of var to be letter*.

The three echo commands demonstrate three levels of expansion. When $var is quoted with single quotation marks, the shell performs no expansion and passes the character string $var to echo, which displays it. When you use double quotation marks, the shell performs variable expansion only and substitutes the value of the var variable for its name, preceded by a dollar sign. No pathname expansion is performed on this command because double quotation marks suppress it. In the final command, the shell, without the limitations of quotation marks, performs variable substitution and then pathname expansion before passing the arguments to echo.

$ ls letter*
letter1 letter2 letter3
$ var=letter*
$ set | grep var
var='letter*
$ echo '$var'
$var
$ echo "$var"
letter*
$ echo $var
letter1 letter2 letter3

Process Substitution

A special feature of the Bourne Again Shell is the ability to replace filename arguments with processes. An argument with the syntax <(command) causes command to be executed and the output written to a named pipe (FIFO). The shell replaces that argument with the name of the pipe. If that argument is then used as the name of an input file during processing, the output of command is read. Similarly an argument with the syntax >(command) is replaced by the name of a pipe that command reads as standard input.

The following example uses sort (page 837) with the m (merge, which works correctly only if the input files are already sorted) option to combine two word lists into a single list. Each word list is generated by a pipe that extracts words matching a pattern from a file and sorts the words in that list.

$ sort -m -f <(grep "[^A-Z]..$" memo1 | sort) <(grep ".*aba.*" memo2 |sort)


Chapter Summary

The shell is both a command interpreter and a programming language. As a command interpreter, the shell executes commands you enter in response to its prompt. As a programming language, the shell executes commands from files called shell scripts. When you start a shell, it typically runs one or more startup files.

Running a shell script

Assuming that the file holding a shell script is in the working directory, there are three basic ways to execute the shell script from the command line.

  1. Type the simple filename of the file that holds the script.

  2. Type a relative pathname, including the simple filename preceded by ./.

  3. Type bash or tcsh followed by the name of the file.

Technique 1 requires that the working directory be in the PATH variable. Techniques 1 and 2 require that you have execute and read permission for the file holding the script. Technique 3 requires that you have read permission for the file holding the script.

Job control

A job is one or more commands connected by pipes. You can bring a job running in the background into the foreground by using the fg builtin. You can put a foreground job into the background by using the bg builtin, provided that you first suspend the job by pressing the suspend key (typically CONTROL-Z). Use the jobs builtin to see which jobs are running or suspended.

Variables

The shell allows you to define variables. You can declare and initialize a variable by assigning a value to it; you can remove a variable declaration by using unset. Variables are local to a process unless they are exported using the export (bash) or setenv (tcsh) builtin to make them available to child processes. Variables you declare are called user-created variables. The shell also defines called keyword variables. Within a shell script you can work with the command line (positional) parameters the script was called with.

Process

Each process has a unique identification (PID) number and is the execution of a single Mac OS X command. When you give it a command, the shell forks a new (child) process to execute the command, unless the command is built into the shell (page 138). While the child process is running, the shell is in a state called sleep. By ending a command line with an ampersand (&), you can run a child process in the background and bypass the sleep state so that the shell prompt returns immediately after you press RETURN. Each command in a shell script forks a separate process, each of which may in turn fork other processes. When a process terminates, it returns its exit status to its parent process. An exit status of zero signifies success and nonzero signifies failure.

History

The history mechanism, a feature adapted from the C Shell, maintains a list of recently issued command lines, also called events, that provides a way to reexecute previous commands quickly. There are several ways to work with the history list; one of the easiest is to use a command line editor.

Command line editors

When using an interactive Bourne Again Shell, you can edit your command line and commands from the history file, using either of the Bourne Again Shell's command line editors (vi[m] or emacs). When you use the vi(m) command line editor, you start in Input mode, unlike the way you normally enter vi(m). You can switch between Command and Input modes. The emacs editor is modeless and distinguishes commands from editor input by recognizing control characters as commands.

Aliases

An alias is a name that the shell translates into another name or (complex) command. Aliases allow you to define new commands by substituting a string for the first token of a simple command. The Bourne Again and TC Shells use different syntaxes to define an alias, but aliases in both shells work similarly.

Functions

A shell function is a series of commands that, unlike a shell script, are parsed prior to being stored in memory so that they run faster than shell scripts. Shell scripts are parsed at runtime and are stored on disk. A function can be defined on the command line or within a shell script. If you want the function definition to remain in effect across login sessions, you can define it in a startup file. Like the functions of a programming language, a shell function is called by giving its name followed by any arguments.

Shell features

There are several ways to customize the shell's behavior. You can use options on the command line when you call bash and you can use the bash set and shopt builtins to turn features on and off.

Command line expansion

When it processes a command line, the Bourne Again Shell may replace some words with expanded text. Most types of command line expansion are invoked by the appearance of a special character within a word (for example, a leading dollar sign denotes a variable). See Table 8-6 on page 292 for a list of special characters. The expansions take place in a specific order. Following the history and alias expansions, the common expansions are parameter and variable expansion, command substitution, and pathname expansion. Surrounding a word with double quotation marks suppresses all types of expansion except parameter and variable expansion. Single quotation marks suppress all types of expansion, as does quoting (escaping) a special character by preceding it with a backslash.


Exercises

1.

Explain the following unexpected result:

$ whereis date
/bin/date
$ echo $PATH
.:/usr/local/bin:/usr/bin:/bin
$ cat > date
echo "This is my own version of date."
$ date
Tue May 24 11:45:49 PDT 2005

2.

What are two ways you can execute a shell script when you do not have execute access permission for the file containing the script? Can you execute a shell script if you do not have read access permission for the file containing the script?

3.

What is the purpose of the PATH variable?

  1. Set the PATH variable so that it causes the shell to search the following directories in order:

    • /usr/local/bin

    • /usr/bin/X11

    • /usr/bin

    • /bin

    • /Developer/Tools

    • The bin directory in your home directory

    • The working directory

  2. If there is a file named doit in /usr/bin and another file with the same name in your ~/bin, which one will be executed? (Assume that you have execute permission for both files.)

  3. If your PATH variable is not set to search the working directory, how can you execute a program located there?

  4. Which command can you use to add the directory /usr/sbin to the end of the list of directories in PATH?

4.

Assume that you have made the following assignment:

$ person=jenny

Give the output of each of the following commands:

  1. echo $person

  2. echo '$person'

  3. echo "$person"

5.

The following shell script adds entries to a file named journal-file in your home directory. This script helps you keep track of phone conversations and meetings.

$ cat journal
# journal: add journal entries to the file
# $HOME/journal-file

file=$HOME/journal-file
date >> $file
echo -n "Enter name of person or group:"
read name
echo "$name" >> $file
echo >> $file
cat >> $file
echo "----------------------------------------------------" >> $file
echo >> $file

  1. What do you have to do to the script to be able to execute it?

  2. Why does the script use the read builtin (page 571) the first time it accepts input from the terminal and the cat utility the second time?

6.

Assume that the /Users/jenny/grants/biblios and /Users/jenny/biblios directories exist. Give Jenny's working directory after she executes each sequence of commands given. Explain what happens in each case.

  1. $ pwd
/Users/jenny/grants
$ CDPATH=$(pwd)
$ cd
$ cd biblios

  2. $ pwd
/Users/jenny/grants
$ CDPATH=$(pwd)
$ cd $HOME/biblios

7.

Name two ways you can identify the PID number of your login shell.

8.

Give the following command:

$ sleep 30 | cat /etc/weekly

Is there any output from sleep? Where does cat get its input from? What has to happen before the shell displays another prompt?


Advanced Exercises

9.

Write a sequence of commands or a script that demonstrates that variable expansion occurs before pathname expansion.

10.

Write a shell script that outputs the name of the shell that is executing it.

11.

Explain the behavior of the following shell script:

$ cat quote_demo
twoliner="This is line 1.
This is line 2."
echo "$twoliner"
echo $twoliner

  1. How many arguments does each echo command see in this script? Explain.

  2. Redefine the IFS shell variable so that the output of the second echo is the same as the first.

12.

Add the exit status of the previous command to your prompt so that it behaves similarly to the following:

$ [0] ls xxx
ls: xxx: No such file or directory
$ [1]

13.

The dirname utility treats its argument as a pathname and writes to standard output the path prefixthat is, everything up to but not including the last component:

$ dirname a/b/c/d
a/b/c

If you give dirname a simple filename (no / characters) as an argument, dirname writes a . to standard output:

$ dirname simple
.

Implement dirname as a bash function. Make sure that it behaves sensibly when given such arguments as /.

14.

Implement the basename utility, which writes the last component of its pathname argument to standard output, as a bash function. For example, given the pathname a/b/c/d, basename writes d to standard output:

$ basename a/b/c/d
d

15.

The Mac OS X basename utility has an optional second argument. If you give the command basename path suffix, basename removes the suffix and the prefix from path:

$ basename src/shellfiles/prog.bash .bash
prog
$ basename src/shellfiles/prog.bash .c
prog.bash

Add this feature to the function you wrote for exercise 14.


Chapter 9. The TC Shell

Previous Page
Next Page

9. The TC Shell

IN THIS CHAPTER

Shell Scripts | 338
Entering and Leaving the TC Shell | 338
Features Common to the Bourne Again and TC Shells | 341
Redirecting Standard Error | 346
Word Completion | 348
Editing the Command Line | 350
Variables | 352
Reading User Input | 358
Control Structures | 364
Builtins | 373

The TC Shell (tcsh) performs the same function as the Bourne Again Shell and other shells: It provides an interface between you and the Mac OS X operating system. The TC Shell is an interactive command interpreter as well as a high-level programming language. Although you use only one shell at any given time, you should be able to switch back and forth comfortably between shells as the need arises (you may want to run different shells in different windows). Chapters 8 and 13 apply to tcsh as well as to bash so they provide a good background for this chapter. This chapter explains tcsh features that are not found in bash and those that are implemented differently from their bash counterparts. The tcsh home page is www.tcsh.org.

The TC Shell is an expanded version of the C Shell (csh), which originated on Berkeley UNIX. The "T" in TC Shell comes from the TENEX and TOPS-20 operating systems, which inspired command completion and other features in the TC Shell. A number of features not found in csh are present in tcsh, including file and username completion, command line editing, and spelling correction. As with csh, you can customize tcsh to make it more tolerant of mistakes and easier to use. By setting the proper shell variables, you can have tcsh warn you when you appear to be accidentally logging out or overwriting a file. Many popular features of the original C Shell are now shared by bash and tcsh.

Assignment statement

Although some of the functionality of tcsh is present in bash, differences arise in the syntax of some commands. For example, the tcsh assignment statement has the following syntax:

set variable = value

Having SPACEs on either side of the equal sign, although illegal in bash, is allowed in tcsh. By convention shell variables in tcsh are generally named with lowercase letters, not uppercase (you can use either). If you reference an undeclared variable (one that has had no value assigned to it), tcsh will give you an error message, whereas bash will not. Finally the default tcsh prompt is a greater than sign (>), but it is frequently set to a single $ character followed by a SPACE. The examples in this chapter use a prompt of tcsh $ to avoid confusion with the bash prompt.

Tip: Do not use tcsh as a programming language

If you have used UNIX and are comfortable with the C or TC Shell, you may want to use tcsh as your login shell. However, you may find that the TC Shell is not as good a programming language as bash. If you are going to learn only one shell programming language, learn bash. The Bourne Again Shell is used throughout Mac OS X to program many system administration scripts.


Shell Scripts

With tcsh you can execute files containing TC Shell commands, just as bash can execute files containing Bourne Again Shell commands. The concepts of writing and executing scripts in the two shells are similar. However, the methods of declaring and assigning values to variables and the syntax of control structures are different.

You can run bash and tcsh scripts while using any one of the shells as a command interpreter. Various methods exist for selecting the shell that runs a script. Refer to "#! Specifies a Shell" on page 267 for more information.

If the first character of a shell script is a pound sign (#) and the following character is not an exclamation point (!), the TC Shell executes the script under tcsh. If the first character is anything other than #, tcsh calls the sh copy of bash to execute the script.


Entering and Leaving the TC Shell

You can execute tcsh by giving the command tcsh. If you are not sure which shell you are using, use the ps utility to find out. It shows whether you are running tcsh, bash, sh (a copy of bash), or possibly another shell. The finger command followed by your username displays the name of your login shell, which is stored in the passwd database. If you want to use tcsh as a matter of course, you can use the chsh (change shell) utility (page 448) to change your login shell. The shell you specify will be in effect for your next login and all subsequent logins until you specify a different login shell.

Tip: Shell game

When you are working with an interactive TC Shell, if you run a script in which # is not the first character of the script and you call the script directly (without preceding its name with tcsh), tcsh calls the sh copy of bash to run the script. The following script was written to be run under tcsh but, when called from a tcsh command line, is executed by bash. The set builtin (page 568) works differently under bash and tcsh. As a result the following example (from page 358) issues a prompt but does not wait for you to respond:

tcsh $ cat user_in
echo -n "Enter input: "
set input_line = "$<"
echo $input_line

tcsh $ user_in
Enter input:

Although in each case the examples are run from a tcsh command line, the following one calls tcsh explicitly so that tcsh executes the script and it runs correctly.

tcsh $ tcsh user_in
Enter input: here is some input
here is some input

Tip: echo: getting rid of the RETURN

The tcsh echo builtin accepts either a n option or a trailing \c to get rid of the RETURN that echo normally displays at the end of a line. The bash echo builtin accepts only the n option (refer to "read: Accepts User Input" on page 571).

You can leave tcsh in several ways. The approach you choose depends on two factors: whether the shell variable ignoreeof is set and whether you are using the shell that you logged in on (your login shell) or another shell that you created after you logged in. If you are not sure how to exit from tcsh, press CONTROL-D on a line by itself with no leading SPACEs, just as you would to terminate standard input to another program. You will either exit or receive instructions on how to exit. If you have not set ignoreeof (page 363) and it has not been set for you in a startup file, you can exit from any shell by using CONTROL-D (the same procedure you use to exit from the Bourne Again Shell).

When ignoreeof is set, CONTROL-D does not work. The ignoreeof variable causes the shell to display a message telling you how to exit. You can always exit from tcsh by giving an exit command. A logout command allows you to exit from your login shell only.

Startup Files

When you log in on the TC Shell, it automatically executes various startup files. These files are normally executed in the order described in this section, but you can compile tcsh so that it uses a different order. You must have read access to a startup file to execute the commands in it.

/etc/csh.cshrc and /etc/csh.login

The shell first executes the commands in /etc/csh.cshrc and /etc/csh.login. Superuser can set up these files to establish systemwide default characteristics for tcsh users. They contain systemwide configuration information, such as the default path, the location to check for mail, and so on.

.tcshrc and .cshrc

Next the shell looks for ~/.tcshrc or, if it does not exist, ~/.cshrc (~/ is shorthand for your home directory). You can use these files to establish variables and parameters that are local to your shell. Each time you create a new shell, tcsh reinitializes these variables for the new shell. The following .tcshrc file sets several shell variables, establishes two aliases (page 344), and adds two new directories to pathone at the start of the list and one at the end:

tcsh $ cat ~/.tcshrc
set noclobber
set dunique
set ignoreeof
set history=256
set path = (~/bin $path /Developer/Tools)
alias h history
alias ll ls -l

.history

Login shells rebuild the history list from the contents of ~/.history. If the histfile variable exists, tcsh uses the file that histfile points to in place of .history.

.login

Login shells read and execute the commands in ~/.login. This file contains commands that you want to execute once, at the beginning of each session. You can use setenv (page 353) to declare environment (global) variables here. You can also declare the type of terminal you are using and set some terminal characteristics in your .login file.

tcsh $ cat ~/.login
setenv history 200
setenv mail /var/spool/mail/$user
if ( -z $DISPLAY ) then
       setenv TERM vt100
   else
       setenv TERM xterm
endif
stty erase '^h' kill '^u' -lcase tab3
date '+Login on %A %B %d at %I:%M %p'

The preceding .login file establishes the type of terminal you are using by setting the TERM variable (the if statement [page 364] checks to see whether it is running under X11 [the DISPLAY variable is set under X11], to determine what value to assign to TERM). It then runs stty (page 854) to set terminal characteristics and date (page 701) to display the time you logged in.

/etc/csh.logout and .logout

The TC Shell runs the /etc/csh.logout and ~/.logout files, in that order, when you exit from a login shell. The following sample .logout file uses date to display the time you logged out. The sleep command ensures that echo has time to display the message before the system logs you out. The delay may be useful for dial-up lines that take some time to display the message.

tcsh $ cat ~/.logout
date '+Logout on %A %B %d at %I:%M %p'
sleep 5


Features Common to the Bourne Again and TC Shells

Most of the features common to both bash and tcsh are derived from the original C Shell:

  • Command line expansion (also called substitution; page 341)

  • History (page 342)

  • Aliases (page 344)

  • Job control (page 345)

  • Filename substitution (page 346)

  • Directory stack manipulation (page 346)

  • Command substitution (page 346)

Because the chapters on bash discuss these features in detail, this section focuses on the differences between the bash and tcsh implementations.

Command Line Expansion (Substitution)

Refer to "Processing the Command Line" on page 321 for an introduction to command line expansion in the Bourne Again Shell. The tcsh man page uses the term substitution instead of expansion, which is used by bash. The TC Shell scans each token for possible expansion in the following order:

  1. History substitution (page 342)

  2. Alias substitution (page 344)

  3. Variable substitution (page 353)

  4. Command substitution (page 346)

  5. Filename substitution (page 346)

  6. Directory stack substitution (page 346)

History

The TC Shell assigns a sequential event number to each command line. You can display this event number as part of the tcsh prompt (refer to "prompt" on page 360). Examples in this section show numbered prompts when they help illustrate the behavior of a command.

history Builtin

As in bash, the tcsh history builtin displays the events in your history list. The list of events is ordered with the oldest events at the top. The last event in the history list is the history command that displayed the list. In the following history list, which is limited to ten lines by the argument of 10 to the history command, command 23 modifies the tcsh prompt to display the history event number. The time each command was executed appears to the right of the event number.

32 $ history 10
    23  23:59   set prompt = "! $ "
    24  23:59   ls -l
    25  23:59   cat temp
    26  0:00    rm temp
    27  0:00    vim memo
    28  0:00    lpr memo
    29  0:00    vim memo
    30  0:00    lpr memo
    31  0:00    rm memo
    32  0:00    history

History Expansion

The same event and word designators work in both shells. For example, !! refers to the previous event in tcsh, just as it does in bash. The command !328 executes event number 328 and !?txt? executes the most recent event containing the string txt. For more information refer to "Using an Exclamation Point (!) to Reference Events" on page 299. Table 9-1 lists the few tcsh word modifiers not found in bash.

Table 9-1. Word modifiers

Modifier

Function

u

Converts the first lowercase letter into uppercase

l

Converts the first uppercase letter into lowercase

a

Applies the next modifier globally within a single word

You can use more than one word modifier in a command. For instance, the a modifier, in combination with the u or l modifier, enables you to change the case of an entire word.

tcsh $ echo $VERSION
VERSION: Undefined variable.
tcsh $ echo !!:1:al
echo $version
tcsh 6.12.00 (Astron) 2002-07-23 (powerpc-apple-darwin) options 8b,...

In addition to using event designators to access the history list, you can use the command line editor to access, modify, and execute previous commands (page 350).

Variables

The variables that you set to control history in tcsh are different from those used in bash. Whereas bash uses HISTSIZE and HISTFILESIZE to determine the number of events that are preserved during and between sessions, tcsh uses history and savehist Table 9-2 for these purposes.

Table 9-2. History variables

Function

Variable

Default

Maximum number of events saved during a session

history

100 events

Location of the history file

histfile

~/.history

Maximum number of events saved between sessions

savehist

not set

history and savehist

When you exit from a tcsh shell, the most recently executed commands are saved in your ~/.history file. The next time you start the shell this file initializes the history list. The value of the savehist variable determines the number of lines saved in the .history file (not necessarily the same as the history variable). If savehist is not set, tcsh does not save history between sessions. The history and savehist variables must be local (declared with set, not setenv). The history variable holds the number of events remembered during a session and the savehist variable holds the number remembered between sessions. See Table 9-2.

If you set the value of history too high, it can use too much memory. If it is unset or set to zero, the shell does not save any commands. To establish a history list of the 500 most recent events, give the following command manually or place it in your ~/.tcshrc startup file:

tcsh $ set history = 500

The following command causes tcsh to save the 200 most recent events across login sessions:

tcsh $ set savehist = 200

You can combine these two assignments into a single command:

tcsh $ set history=500 savehist=200

After you set savehist you can log out and log in again, and the 200 most recent events from the previous login sessions will appear in your history list. Set savehist in your ~/.tcshrc file if you want to maintain your event list from login to login.

histlit

If you set the variable histlit (history literal), history displays the commands in the history list exactly as they were typed in without any shell interpretation. The following example shows the effect of this variable (compare the lines numbered 32):

tcsh $ cat /etc/csh.cshrc
...
tcsh $ cp !!:1 ~
cp /etc/csh.cshrc ~
tcsh $ set histlit
tcsh $ history
...
   31  9:35    cat /etc/csh.cshrc
   32  9:35    cp !!:1 ~
   33  9:35    set histlit
   34  9:35    history
tcsh $ unset histlit
tcsh $ history
...
   31  9:35    cat /etc/csh.cshrc
   32  9:35    cp /etc/csh.cshrc ~
   33  9:35    set histlit
   34  9:35    history
   35  9:35    unset histlit
   36  9:36    history

Optional

There is a difference in how bash and tcsh expand history event designators. If you give the command !250w, bash replaces it with command number 250 with a character w appended to it. In contrast, tcsh looks back through your history list for an event that begins with the string 250w to execute. The reason for the difference: bash interprets the first three characters of 250w as the number of a command, whereas tcsh interprets those characters as part of the search string 250w. (If the 250 stands alone, tcsh TReats it as a command number.)

If you want to append w to command number 250, you can insulate the event number from the w by surrounding it with braces:

!{250}w

Aliases

The alias/unalias feature in tcsh closely resembles its counterpart in bash (page 311). However, the alias builtin has a slightly different syntax:

alias name value

The following command creates an alias for ls:

tcsh $ alias ls "ls -lF"

The tcsh alias lets you substitute command line arguments, whereas bash does not:

$ alias nam "echo Hello, \!^ is my name"
$ nam Sam
Hello, Sam is my name

The string \!* within an alias expands to all command line arguments:

$ alias sortprint "sort \!* lpr"

The next alias displays its second argument:

$ alias n2 "echo \!:2"

Special Aliases

Some alias names, called special aliases, have special meaning to tcsh. If you define an alias with one of these names, tcsh executes it automatically as explained in Table 9-3. Initially all special aliases are undefined.

Table 9-3. Special aliases

Alias

When executed

beepcmd

Whenever the shell would normally ring the terminal bell. Gives you a way to have other visual or audio effects take place at those times.

cwdcmd

Whenever you change to another working directory.

periodic

Periodically, as determined by the number of minutes in the tperiod variable. If tperiod is unset or has the value 0, periodic has no meaning.

precmd

Just before the shell displays a prompt.

shell

Gives the absolute pathname of the shell that you want to use to run scripts that do not start with #! (page 267).

To see a list of current aliases, give the command alias. To view the alias for a particular name, give the command alias followed by the name.

History Substitution in Aliases

You can substitute command line arguments by using the history mechanism, where a single exclamation point represents the command line containing the alias. Modifiers are the same as those used by history (page 299). In the following example, the exclamation points are quoted so that the shell does not interpret them when building the aliases:

21 $ alias last echo \!:$
22 $ last this is just a test
test
23 $ alias fn2 echo \!:2:t
24 $ fn2 /Users/jenny/test /Users/alex/temp /Users/barbara/new
 temp

Event 21 defines for last an alias that displays the last argument. Event 23 defines for fn2 an alias that displays the simple filename, or tail, of the second argument on the command line.

Job Control

Job control is similar in both bash (page 272) and tcsh. You can move commands between the foreground and background, suspend jobs temporarily, and get a list of the current jobs. The % character references a job when followed by a job number or a string prefix that uniquely identifies the job. You will see a minor difference when you run a multiple-process command line in the background from each shell. Whereas bash displays only the PID number of the last background process in each job, tcsh displays the numbers for all processes belonging to a job. The example from page 272 looks like this under tcsh:

tcsh $ find . -print | sort | lpr & grep -l alex /tmp/* > alexfiles &
[1] 18839 18840 18841
[2] 18876

Filename Substitution

The TC Shell expands the characters *, ?, and [] in a pathname just as bash does (page 133). The * matches any string of zero or more characters, ? matches any single character, and [] defines a character class, which is used to match single characters appearing within a pair of brackets.

The TC Shell expands command line arguments that start with a tilde (~) into filenames in much the same way that bash does (page 348), with the ~ standing for the user's home directory or the home directory of the user whose name follows the tilde. The bash special expansions ~+ and ~ are not available in tcsh.

Brace expansion (page 323) is available in tcsh. Like tilde expansion, it is regarded as an aspect of filename substitution even though brace expansion can generate strings that are not the names of actual files.

In tcsh and its predecessor csh, the process of using patterns to match filenames is referred to as globbing and the pattern itself is called a globbing pattern. If tcsh is unable to identify one or more files that match a globbing pattern, it reports an error (unless the pattern contains a brace). Setting the shell variable noglob suppresses filename substitution, including both tilde and brace interpretation.

Manipulating the Directory Stack

Directory stack manipulation in tcsh does not differ much from that in bash (page 275). The dirs builtin displays the contents of the stack, and the pushd and popd builtins push directories onto and pop directories off of the stack.

Command Substitution

The $(...) format for command substitution is not available in tcsh. In its place you must use the original '...' format. Otherwise, the implementation in bash and tcsh is identical. Refer to page 327 for more information on command substitution.


Redirecting Standard Error

Both bash and tcsh use a greater than symbol (>) to redirect standard output, but tcsh does not use the bash notation 2> to redirect standard error. Under tcsh you use a greater than symbol followed by an ampersand (>&) to combine and redirect standard output and standard error. Although you can use this notation under bash, it is not common. The following examples, like the bash examples on page 263, reference file x, which does not exist, and file y, which contains a single line.

tcsh $ cat x
cat: x: No such file or directory
tcsh $ cat y
This is y.
tcsh $ cat x y >& hold
tcsh $ cat hold
cat: x: No such file or directory
This is y.

With an argument of y in the preceding example, cat sends a string to standard output. An argument of x causes cat to send an error message to standard error.

Unlike bash, tcsh does not provide a simple way to redirect standard error separately from standard output. A work-around frequently provides a reasonable solution. The following example runs cat with arguments of x and y in a subshell (the parentheses ensure that the command within them runs in a subshellsee page 271). Also within the subshell a > redirects standard output to the file outfile. Output sent to standard error is not touched by the subshell but rather is sent to the parent shell, where both it and standard output are sent to errfile. Because standard output has already been redirected, errfile contains only output sent to standard error.

tcsh $ (cat x y > outfile) >& errfile
tcsh $ cat outfile
This is y.
tcsh $ cat errfile
cat: x: No such file or directory

It can be useful to combine and redirect output when you want to run a slow command in the background and do not want its output cluttering up the terminal screen. For example, because the find utility (page 728) often takes some time to complete, it may be a good idea to run it in the background. The next command finds in the filesystem hierarchy all files that contain the string biblio in their name. The command runs in the background and sends its output to the findout file. Because the find utility sends to standard error a report of directories that you do not have permission to search, the findout file contains a record of any files that are found as well as a record of the directories that could not be searched.

tcsh $ find / -name "*biblio*" -print >& findout &

In this example, if you did not combine standard error with standard output and redirected only standard output, the error messages would appear on the screen and findout would list only files that were found.

While a command that has its output redirected to a file is running in the background, you can look at the output by using tail (page 859) with the f option. The f option causes tail to display new lines as they are written to the file:

tcsh $ tail -f findout

To terminate the tail command, press the interrupt key (usually CONTROL-C).


Working with the Command Line

This section covers word completion, editing the command line, and correcting spelling.

Word Completion

The TC Shell completes filenames, commands, and variable names on the command line when you prompt it to do so. The generic term used to refer to all these features under tcsh is word completion.

Filename Completion

The TC Shell can complete a filename after you specify a unique prefix. Filename completion is similar to filename generation, but the goal of filename completion is to select a single file. Together they make it practical to use long, descriptive filenames.

To use filename completion when you are entering a filename on the command line, type enough of the name to identify the file in the directory uniquely and press TAB; tcsh fills in the name and adds a SPACE, leaving the cursor so you can enter additional arguments or press RETURN. In the following example, the user types the command cat trig1A and presses TAB; the system fills in the rest of the filename that begins with trig1A:

tcsh $ cat trig1A    TAB    cat trig1A.302488

If two or more filenames match the prefix that you have typed, tcsh cannot complete the filename without obtaining more information from you. The shell attempts to maximize the length of the prefix by adding characters, if possible, and then beeps to signify that additional input is needed to resolve the ambiguity:

tcsh $ ls h*
help.hist help.trig01 help.txt
tcsh $ cat h    TAB    cat help. (beep)

You can fill in enough characters to resolve the ambiguity and then press TAB again. Alternatively, you can press CONTROL-D to cause tcsh to display a list of matching filenames:

tcsh $ cat help.    CONTROL-D
help.hist help.trig01 help.txt
tcsh $ cat help.

After displaying the filenames tcsh redraws the command line so you can disambiguate the filename (and press TAB again) or finish typing the filename manually.

Tilde Completion

The TC Shell parses a tilde (~) appearing as the first character of a word and attempts to expand it to a username when you enter a TAB:

tcsh $ cd ~al    TAB    cd ~alex/      RETURN
tcsh $ pwd
/Users/alex

By appending a slash (/), tcsh indicates that the completed word is a directory. The slash also makes it easy to continue specifying the pathname.

Command and Variable Completion

You can use the same mechanism that you use to list and complete filenames with command and variable names. Unless you give a full pathname, the shell uses the variable path in an attempt to complete a command name. The choices listed are likely to be located in different directories.

tcsh $ up    TAB (beep)    CONTROL-D
update                 update_prebinding_core
update_prebinding      uptime
tcsh $ up    t    TAB    uptime       RETURN
 9:59am up 31 days, 15:11, 7 users, load averages: 0.03, 0.02, 0.00

If you set the autolist variable as in the following example, the shell lists choices automatically when you invoke completion by pressing TAB. You do not have to press CONTROL-D.

tcsh $ set autolist
tcsh $ up    TAB (beep)
update                   update_prebinding_core
update_prebinding        uptime
tcsh $ up    t    TAB    uptime       RETURN
 10:01am up 31 days, 15:14, 7 users, load averages: 0.20, 0.06, 0.02

If you set autolist to ambiguous, the shell lists the choices when you press TAB only if the word you enter is the longest prefix of a set of commands. Otherwise, pressing TAB causes the shell to add one or more characters to the word until it is the longest prefix; pressing TAB again then lists the choices:

tcsh $ set autolist=ambiguous
tcsh $ echo $h    TAB (beep)
histfile history home
tcsh $ echo $h      i    TAB    echo $hist      TAB
histfile history
tcsh $ echo $hist      o    TAB    echo $history       RETURN
1000

The shell must rely on the context of the word within the input line to determine whether it is a filename, a username, a command, or a variable name. The first word on an input line is assumed to be a command name; if a word begins with the special character $, it is viewed as a variable name; and so on. In the following example, the second which command does not work properly: The context of the word up makes it look like the beginning of a filename rather than the beginning of a command. The TC Shell supplies which with an argument of updates (a nonexecutable file) and which displays an error message:

tcsh $ ls up*
updates
tcsh $ which uptime
/usr/bin/uptime
tcsh $ which up    TAB    which updates
updates: Command not found.

Editing the Command Line

bindkey

The tcsh command line editing feature is similar to that available under bash. You can use either emacs mode commands (default) or vi(m) mode commands. Change to vi(m) mode commands by using bindkey v and to emacs mode commands by using bindkey e. The ARROW keys are bound to the obvious motion commands in both modes, so you can move back and forth (up and down) through your history list as well as left and right on the current command line.

Without pan argument, the bindkey builtin displays the current mappings between editor commands and the key sequences you can enter at the keyboard:

tcsh $ bindkey
Standard key bindings
"^@"        ->  set-mark-command
"^A"        ->  beginning-of-line
"^B"        ->  backward-char
"^C"        ->  tty-sigintr
"^D"        ->  delete-char-or-list-or-eof
"^E"        ->  end-of-line
...
Alternative key bindings
Multi-character bindings
"^[[A"      ->  up-history
"^[[B"      ->  down-history
"^[[C"      ->  forward-char
"^[[D"      ->  backward-char
"^[[H"      ->  beginning-of-line
"^[[F"      ->  end-of-line
...
Arrow key bindings
down             ->  down-history
up               ->  up-history
left             ->  backward-char
right            ->  forward-char
home             ->  beginning-of-line
end              ->  end-of-line

The ^ indicates a CONTROL character (^B = CONTROL-B). The ^[indicates a META or ALT character; you press and hold the META or OPTION key while you press the key for the next character. If this substitution does not work or if the keyboard you are using does not have a META or OPTION key, press and release the ESCAPE key and then press the key for the next character. If the OPTION key does not work while you are using the Terminal utility, see the tip "Activating the META key" on page 31. For ^[[F you would press META-[ or ALT-[ followed by the F key or else ESCAPE[ F). The down/up/left/right indicate ARROW keys, and home/end indicate the HOME and END keys on the numeric keypad.

The preceding example shows the output from bindkey with the user in emacs mode. Change to vi(m) mode (bindkey v) and give another bindkey command to display the vi(m) key bindings. You can pipe the output of bindkey through less to make it easier to read the list.

Correcting Spelling

You can have tcsh attempt to correct the spelling of command names, filenames, and variables (but only using emacs-style key bindings). Spelling correction can take place only at two times: before and after you press RETURN.

Before You Press RETURN

For tcsh to correct a word or line before you press RETURN, you must indicate that you want it to do so. The two functions for this purpose are spell-line and spell-word:

$ bindkey | grep spell
"^[$"        -> spell-line
"^[S"        -> spell-word
"^[s"        -> spell-word

The output from bindkey shows that spell-line is bound to META-$ (ALT-$ or ESCAPE $) and spell-word is bound to META-S and META-s (ALT-s or ESCAPE s and ALT-S or ESCAPE S). To correct the spelling of the word to the left of the cursor, enter META-s. Entering META-$ invokes the spell-line function, which attempts to correct all words on a command line:

tcsh $ ls
bigfile.gz
tcsh $ gunzipp    META-s    gunzip bigfele.gz    META-s    gunzip bigfile.gz
tcsh $ gunzip bigfele.gz    META-$    gunzip bigfile.gz
tcsh $ ecno $usfr    META-$    echo $user

After You Press RETURN

The variable named correct controls what tcsh attempts to correct or complete after you press RETURN and before it passes the command line to the command being called. If you do not set correct, tcsh will not correct anything:

tcsh $ unset correct
tcsh $ ls morning
morning
tcsh $ ecno $usfr morbing
usfr: Undefined variable.

The shell reports the error in the variable name and not the command name because it expands variables before it executes the command (page 341). When you give a bad command name without any arguments, the shell reports on the bad command name.

Set correct to cmd to correct only commands; all to correct commands, variables, and filenames; or complete to complete commands:

tcsh $ set correct = cmd
tcsh $ ecno $usfr morbing

CORRECT>echo $usfr morbing (y|n|e|a)? y
usfr: Undefined variable.

tcsh $ set correct = all
tcsh $ ecno $usfr morbing

CORRECT>echo $user morning (y|n|e|a)? y
alex morning

With correct set to cmd, tcsh corrects the command name from ecno to echo. With correct set to all, tcsh corrects both the command name and the variable. It would also correct a filename if one was present on the command line.

Automatic spell checking displays a special prompt that lets you enter y to accept the modified command line, n to reject it, e to edit it, or a to abort the command. Refer to "prompt3" on page 360 for a discussion of the special prompt used in spelling correction.

In the next example, after setting the correct variable the user mistypes the name of the ls command; tcsh then prompts for a correct command name. Because the command that tcsh has offered as a replacement is not ls, the user chooses to edit the command line. The shell leaves the cursor following the command so the user can correct the mistake:

tcsh $ set correct=cmd
tcsh $ lx -l    RETURN (beep)
CORRECT>lex -l (y|n|e|a)? e
tcsh $ lx -l

If you assign the value complete to the variable correct, tcsh attempts command name completion in the same manner as filename completion (page 348). In the following example, after setting correct to complete the user enters the command up. The shell responds with Ambiguous command because several commands start with these two letters but differ in the third letter. The shell then redisplays the command line. The user could press TAB at this point to get a list of commands that start with up but decides to enter t and press RETURN. The shell completes the command because these three letters uniquely identify the uptime utility:

tcsh $ set correct = complete
tcsh $ upRETURN
Ambiguous command
tcsh $ up    tRETURN    uptime
 4:45pm up 5 days, 9:54, 5 users, load averages: 1.62, 0.83, 0.33


Variables

Although tcsh stores variable values as strings, you can work with these variables as numbers. Expressions in tcsh can use arithmetic, logical, and conditional operators. The @ builtin can evaluate integer arithmetic expressions.

This section uses the term numeric variable to describe a string variable that contains a number that tcsh uses in arithmetic or logical arithmetic computations. However, no true numeric variables exist in tcsh.

Variable name

A tcsh variable name consists of 1 to 20 characters, which can be letters, digits, and underscores (_). The first character cannot be a digit but can be an underscore.

Variable Substitution

Three builtins declare, display, and assign values to variables: set, @, and setenv. The set and setenv builtins both assume nonnumeric string variables. The @ builtin works only with numeric variables. Both set and @ declare local variables. The setenv builtin declares a variable and places it in the calling environment of all child processes (makes it global). Using setenv is similar to assigning a value to a variable and then using export in the Bourne Again Shell. See "Locality of Variables" on page 560 for a discussion of local and environment variables.

Once the valueor merely the existenceof a variable has been established, tcsh substitutes the value of that variable when the name of the variable, preceded by a dollar sign ($), appears on a command line. If you quote the dollar sign by preceding it with a backslash or enclosing it within single quotation marks, the shell does not perform the substitution. When a variable is within double quotation marks, the substitution occurs even if you quote the dollar sign by preceding it with a backslash.

String Variables

The TC Shell treats string variables similarly to the way the Bourne Again Shell does. The major difference is in their declaration and assignment: tcsh uses an explicit command, set (or setenv), to declare and/or assign a value to a string variable.

tcsh $ set name = fred
tcsh $ echo $name
fred
tcsh $ set
argv    ()
cwd     /Users/alex
home    /Users/alex
name    fred
path    (/bin /sbin /usr/bin /usr/sbin)
prompt  $
shell   /bin/tcsh
status  0
term    vt100
user    alex

The first line in the example declares the variable name and assigns the string fred to it. Unlike bash, tcsh allows but does not demand SPACEs around the equal sign. The next line displays this value. When you give a set command without any arguments, it displays a list of all local shell variables and their values (your list will be longer than the one in the example). When you give a set command with the name of a variable and no value, the command sets the value of the variable to a null string.

You can use the unset builtin to remove a variable:

tcsh $ set name
tcsh $ echo $name

tcsh $ unset name
tcsh $ echo $name
name: Undefined variable.

With setenv you must separate the variable name from the string being assigned to it by one or more SPACEs and no equal sign. The tcsh command creates a subshell, echo shows that the variable and its value are known to the subshell, and exit returns to the original shell. Try this example, using set in place of setenv:

tcsh $ setenv SCRDIR /usr/local/src
tcsh $ tcsh
tcsh $ echo $SCRDIR
/usr/local/src
tcsh $ exit

If you use setenv with no arguments, it displays a list of the environment (global) variablesvariables that are passed to the shell's child processes. By convention, environment variables are named using uppercase letters.

As with set, giving setenv a variable name without a value sets the value of the variable to a null string. Although you can use unset to remove environment and local variables, unsetenv can remove only environment variables.

Arrays of String Variables

An array is a collection of strings, each of which is identified by its index (1, 2, 3, and so on). Arrays in tcsh use one-based indexing (the first element of the array has the subscript 1). Before you can access individual elements of an array, you must declare the entire array by assigning a value to each element of the array. The list of values must be enclosed in parentheses and separated by SPACEs:

8 $ set colors = (red green blue orange yellow)
9 $ echo $colors
red green blue orange yellow
10 $ echo $colors[3]
blue
11 $ echo $colors[2-4]
green blue orange
12 $ set shapes = ('' '' '' '' '')
13 $ echo $shapes

14 $ set shapes[4] = square
15 $ echo $shapes[4]
square

Event 8 declares the array of string variables named colors to have five elements and assigns values to each of them. If you do not know the values of the elements at the time you declare an array, you can declare an array containing the necessary number of null elements (event 12).

You can reference an entire array by preceding its name with a dollar sign (event 9). A number in brackets following a reference to the array refers to an element of the array (events 10, 14, and 15). Two numbers in brackets, separated by a hyphen, refer to two or more adjacent elements of the array (event 11). Refer to "Special Variable Forms" on page 357 for more information on arrays.

Numeric Variables

The @ builtin assigns the result of a numeric calculation to a numeric variable (as described under "Variables" [page 352], tcsh has no true numeric variables). You can declare single numeric variables with @, just as you can use set to declare nonnumeric variables. However, if you give it a nonnumeric argument, @ displays an error message. Just as set does, the @ command used without any arguments lists all shell variables.

Many of the expressions that the @ builtin can evaluate and the operators it recognizes are derived from the C programming language. The following format shows a declaration or assignment using @ (the SPACE after the @ is required):

@ variable-name operator expression

The variable-name is the name of the variable that you are assigning a value to. The operator is one of the C assignment operators: =, +=, =, *=, /=, or %=. (See page 614 for an explanation of these operators.) The expression is an arithmetic expression that can include most C operators (see the next section). You can use parentheses within the expression for clarity or to change the order of evaluation. Parentheses must surround parts of the expression that contain any of the following characters: <, >, &, or |.

Tip: Do not use $ when assigning a value to a variable

As with bash, variables having a value assigned to them (those on the left of the operator) must not be preceded by a dollar sign ($). Thus

tcsh $ @ $answer = 5 + 5

will yield

answer: Undefined variable.

or, if answer is defined,

@: Variable name must begin with a letter.

whereas

tcsh $ @ answer = 5 + 5

assigns the value 10 to the variable answer.

Expressions

An expression is composed of constants, variables, and most any of the bash operators (page 588). Expressions that involve files rather than numeric variables or strings are described in Table 9-8 on page 365.

Expressions follow these rules:

  1. The shell evaluates a missing or null argument as 0.

  2. All results are decimal numbers.

  3. Except for != and ==, the operators act on numeric arguments.

  4. You must separate each element of an expression from adjacent elements by a SPACE, unless the adjacent element is &, |, <, >, (, or ).

Following are some examples that use @:

216 $ @ count = 0
217 $ echo $count
0
218 $ @ count = ( 10 + 4 ) / 2
219 $ echo $count
7
220 $ @ result = ( $count < 5 )
221 $ echo $result
0
222 $ @ count += 5
223 $ echo $count
12
224 $ @ count++
225 $ echo $count
13

Event 216 declares the variable count and assigns it a value of 0. Event 218 shows the result of an arithmetic operation being assigned to a variable. Event 220 uses @ to assign the result of a logical operation involving a constant and a variable to result. The value of the operation is false (= 0) because the variable count is not less than 5. Event 222 is a compressed form of the following assignment statement:

tcsh $ @ count = $count + 5

Event 224 uses a postfix operator to increment count by 1.

Postincrement and postdecrement operators

You can use the postincrement (++) and postdecrement () operators only in expressions containing a single variable name, as shown in the following example:

tcsh $ @ count = 0
tcsh $ @ count++
tcsh $ echo $count
1
tcsh $ @ next = $count++
@: Badly formed number.

Unlike in the C programming language and bash, expressions in tcsh cannot use preincrement and predecrement operators.

Arrays of Numeric Variables

You must use the set builtin to declare an array of numeric variables before you can use @ to assign values to the elements of that array. The set builtin can assign any values to the elements of a numeric array, including zeros, other numbers, and null strings.

Assigning a value to an element of a numeric array is similar to assigning a value to a simple numeric variable. The only difference is that you must specify the element, or index, of the array. The syntax is

@ variable-name[index] operator expression

The index specifies the element of the array that is being addressed. The first element has an index of 1. The index cannot be an expression but must be either a numeric constant or a variable. In the preceding syntax the brackets around index are part of the syntax and do not indicate that index is optional. If you specify an index that is too large for the array you declared with set, tcsh displays @: Subscript out of range.

226 $ set ages = (0 0 0 0 0)
227 $ @ ages[2] = 15
228 $ @ ages[3] = ($ages[2] + 4)
229 $ echo $ages[3]
19
230 $ echo $ages
0 15 19 0 0
231 $ set index = 3
232 $ echo $ages[$index]
19
233 $ echo $ages[6]
ages: Subscript out of range.

Elements of a numeric array behave as though they were simple numeric variables. Event 226 declares an array with five elements, each having a value of 0. Events 227 and 228 assign values to elements of the array, and event 229 displays the value of one of the elements. Event 230 displays all the elements of the array, 232 specifies an element by using a variable, and 233 demonstrates the out-of-range error message.

Braces

Like with bash, tcsh allows you to use braces to distinguish a variable from surrounding text without the use of a separator:

$ set bb=abc
$ echo $bbdef
bbdef: Undefined variable.
$ echo ${bb}def
abcdef

Special Variable Forms

The special variable with the following syntax has the value of the number of elements in the variable-name array:

$#variable-name

You can determine whether variable-name has been set by looking at the value of the variable with the following syntax:

$?variable-name

This variable has a value of 1 if variable-name is set and 0 otherwise:

tcsh $ set days = (mon tues wed thurs fri)
tcsh $ echo $#days
5
tcsh $ echo $?days
1
tcsh $ unset days
tcsh $ echo $?days
0

Reading User Input

Within a tcsh shell script, you can use the set builtin to read a line from the terminal and assign it to a variable. The following portion of a shell script prompts the user and reads a line of input into the variable input_line:

echo -n "Enter input: "
set input_line = "$<"

The value of the shell variable $< is a line from standard input. The quotation marks around $< keep the shell from assigning only the first word of the line of input to the variable input_line.

Shell Variables

TC Shell variables may be set by the shell, inherited by the shell from the environment, or set by the user and used by the shell. Some variables take on significant values (for example, the PID number of a background process). Other variables act as switches: on if they are declared and off if they are not. Many of the shell variables are often set from one of tcsh's two startup files: ~/.login and ~/.tcshrc (page 340).

Shell Variables That Take on Values

argv

Contains the command line arguments (positional parameters) from the command line that invoked the shell. Like all tcsh arrays, this array uses one-based indexing; argv[1] contains the first command line argument. You can abbreviate references to $argv[n] as $n. The token argv[*] references all the arguments together; you can abbreviate it as $*. Use $0 to reference the name of the calling program. Refer to "Positional Parameters" on page 564. The Bourne Again Shell does not use the argv form, only the abbreviated form.

$#argv or $#

Holds the number of elements in the argv array. Refer to "Special Variable Forms" on page 357.

autolist

Controls command and variable completion (page 349).

autologout

Enables tcsh's automatic logout facility, which logs you out if you leave the shell idle for too long. The value of the variable is the number of minutes of inactivity that tcsh waits before logging you out. The default is 60 minutes if you are Superuser. This variable is initially unset for other users.

cdpath

Affects the operation of cd in the same way as the CDPATH variable does in bash (page 290). The cdpath variable is assigned an array of absolute pathnames (see path, later in this section) and is usually set in the ~/.login file with a command line such as the following:

tcsh $ set cdpath = (/Users/sam /Users/sam/letters)

When you call cd with a simple filename, it searches the working directory for a subdirectory with that name. If one is not found, cd searches the directories listed in cdpath for the subdirectory.

correct

Set to cmd for automatic spelling correction of command names, to all to correct the entire command line, and to complete for automatic completion of command names. This variable works on corrections that are made after you press RETURN. Refer to "After You Press RETURN" on page 351.

cwd

The shell sets this variable to the name of the working directory. When you access a directory through a symbolic link (page 105), tcsh sets cwd to the name of the symbolic link.

dirstack

The shell keeps the stack of directories used with the pushd, popd, and dirs builtins in this variable. For more information refer to "Manipulating the Directory Stack" on page 275.

fignore

Holds an array of suffixes that tcsh ignores during filename completion.]

gid

The shell sets this variable to your group ID.

histfile

Holds the full pathname of the file that saves the history list between login sessions (page 343). The defaults is ~/.history.

history

Specifies the size of your history list. Refer to "History" on page 342.

home or HOME

Holds the pathname of the user's home directory. The cd builtin refers to this variable, as does the filename substitution of ~ (page 324).

mail

Specifies files and directories to check for mail. The TC Shell checks for new mail every 10 minutes unless the first word of mail is a number, in which case that number specifies how often the shell should check in seconds.

owd

The shell keeps the name of your previous (old) working directory in this variable, which is equivalent to ~ in bash.

path or PATH

Holds a list of directories that tcsh searches for executable commands (page 285). If this array is empty or unset, you can execute commands only by giving their full pathnames. You can set path with a command such as the following:

tcsh $ set path = ( /usr/bin /bin /usr/local/bin /usr/bin/X11 ~/bin . )

prompt

Holds the primary prompt, similar to the bash PS1 variable (page 286). If it is not set, the prompt is >, or # for root (Superuser). The shell expands an exclamation point in the prompt string to the current event number. The following is a typical line from a .tcshrc file that sets the value of prompt:

set prompt = '! $ '

Table 9-4 lists a number of special formatting sequences you can use in prompt to achieve special effects.

Table 9-4. prompt formatting sequences

Sequence

Displays in prompt

%/

Value of cwd (the working directory)

%~

Same as %/, but replaces the path of the user's home directory with a tilde

%! or %h or !

Current event number

%m

Hostname without the domain

%M

Full hostname, including the domain

%n

User's username

%t

Time of day through the current minute

%p

Time of day through the current second

%d

Day of the week

%D

Day of the month

%W

Month as mm

%y

Year as yy

%Y

Year as yyyy

%#

A pound sign (#) if the user is running as root (Superuser); otherwise a greater than sign (>)

%?

Exit status of the preceding command

prompt2

Holds the prompt used in foreach and while control structures (pages 369 and 371). The default value is '%R?', where R is replaced by the word while if you are inside a while structure and foreach if you are inside a foreach structure.

prompt3

Holds the prompt used during automatic spelling correction. The default value is 'CORRECT>%R (y|n|e|a)?', where R is replaced by the corrected string.

savehist

Specifies the number of commands saved from the history list when you log out. These events are saved in a file named ~/.history. The shell uses these events as the initial history list when you log in again, causing your history list to continue across login sessions (page 343).

shell

Holds the pathname of the shell you are using.

shlvl

Is incremented each time you start a subshell and decremented each time you exit a subshell. The value is set to 1 for login a shell.

status

Contains the exit status returned by the last command. Similar to $? in bash (page 564).

tcsh

Holds the version number of tcsh that you are running.

time

Provides two functions: automatic timing of commands using the time builtin and the format used by time. You can set this variable to either a single numeric value or an array holding a numeric value and a string. The numeric value is used to control automatic timing; any command that takes more than that number of CPU seconds to run has time display the command statistics when it finishes execution. A value of 0 results in statistics being displayed after every command. The string controls the formatting of the statistics using special formatting sequences, including those listed in Table 9-5.

Table 9-5. time formatting sequences

Sequence

Displays

%U

Time the command spent running user code, in CPU seconds (user mode)

%S

Time the command spent running system code, in CPU seconds (kernel mode)

%E

Wall clock time (total elapsed) taken by the command

%P

Percentage of time the CPU spent on this task during this period, computed as (%U+%S)/%E

%W

Number of times the command's processes were swapped out to disk

%X

Average amount of shared code memory used by the command, in kilobytes

%D

Average amount of data memory used by the command, in kilobytes

%K

Total memory used by the command (as %X+%D), in kilobytes

%M

Maximum amount of memory used by the command, in kilobytes

%F

Number of major page faults (pages of memory that had to be read from disk)

%I

Number of input operations

%O

Number of output operations

By default the time builtin uses the string

"%Uu %Ss %E %P% %X+%Dk %I+%Oio %Fpf+%Ww"

which generates output in the following format:

tcsh $ time
0.200u 0.340s 17:32:33.27 0.0%      0+0k 0+0io 1165pf+0w

You can time commands when you are concerned about system performance. If your commands consistently show many page faults and swaps, your system is probably memory starved and you should consider adding more memory to the system. You can use the information that time reports to compare the performance of various system configurations and program algorithms.

tperiod

Controls how often, in minutes, the shell executes the special periodic alias (page 345).

user

The shell sets this variable to your username.

version

The shell sets this variable to contain detailed information about the version of tcsh you are using.

watch

Set to an array of user and terminal pairs to watch for logins and logouts. The word any means any user or any terminal, so (any any) monitors all logins and logouts on all terminals, and (sam ttyp1 any console $user any) watches for sam on ttyp1, any user who accesses the system console, and any logins and logouts that use your account (presumably to catch intruders). By default logins and logouts are checked once every 10 minutes, but you can change this value by beginning the array with a numeric value giving the number of minutes between checks. If you set watch to (1 any console), logins and logouts by any user on the console will be checked once a minute. Reports are displayed just before a new shell prompt is issued. Also, the log builtin forces an immediate check whenever it is executed. See who for information about how you can control the format of the watch messages.

who

Controls the format of the information displayed in watch messages (Table 9-6).

Table 9-6. who formatting sequence

Sequence

Displays

%n

Username

%a

Action taken by user

%l

Terminal on which action took place

%M

Full hostname of remote host (or local if none) from which action took place

$m

Hostname without domain name

The default string used for watch messages when who is unset is "%n has %a %l from %m", which generates the following line:

jenny has logged on ttyp2 from localhost

$ As in bash, this variable contains the PID number of the current shell; use it as $$.

Shell Variables That Act as Switches

The following shell variables act as switches; their values are not significant. If the variable has been declared, the shell takes the specified action. If not, the action is not taken or is negated. You can set these variables in your ~/.tcshrc startup file, in a shell script, or from the command line.

autocorrect

Causes the shell to attempt spelling correction automatically, just before each attempt at completion.

dunique

Normally pushd blindly pushes the new working directory onto the directory stack, meaning that you can end up with many duplicate entries on this stack. Set dunique to cause the shell to look for and delete any entries that duplicate the one it is about to push.

echo

Causes the shell to display each command before it executes that command. Set echo by calling tcsh with the x option or by using set.

filec

Enables filename completion (page 348) when running tcsh as csh (and csh is linked to tcsh).

histlit

Displays the commands in the history list exactly as entered, without interpretation by the shell (page 344).

ignoreeof

Prevents you from using CONTROL-D to exit from a shell so you cannot accidentally log out. When this variable is declared, you must use exit or logout to leave a shell.

listjobs

Causes the shell to list all jobs whenever a job is suspended.

listlinks

Causes the IsF builtin to show the type of file each symbolic link points to instead of marking the symbolic link with an @ symbol.

loginsh

Set by the shell if the current shell is running as a login shell.

nobeep

Disables all beeping by the shell.

noclobber

Prevents you from accidentally overwriting a file when you redirect output and prevents you from creating a file when you attempt to append output to a nonexistent file (Table 9-7). To override noclobber, add an exclamation point to the symbol you use for redirecting or appending output (for example, >! and >>!). For more information see page 125.

Table 9-7. How noclobber works

Command line

noclobber not declared

noclobber declared

x> fileout

Redirects standard output from process x to fileout. Overwrites fileout if it exists.

Redirects standard output from process x to fileout. The shell displays an error message if fileout exists and does not overwrite the file.

x>> fileout

Redirects standard output from process x to fileout. Appends new output to the end of fileout if it exists. Creates fileout if it does not exist.

Redirects standard output from process x to fileout. Appends new output to the end of fileout if it exists. The shell displays an error message if fileout does not exist and does not create the file.

noglob

Prevents the shell from expanding ambiguous filenames. Allows you to use *, ?, ~, and [] on the command line or in a shell script without quoting them.

nonomatch

Causes the shell to pass an ambiguous file reference that does not match a filename to the command that is being called. The shell does not expand the file reference. When you do not set nonomatch, tcsh generates a No match error message and does not execute the command.

tcsh $ cat questions?
cat: No match
tcsh $ set nonomatch
tcsh $ cat questions?
cat: questions?: No such file or directory

notify

When set, tcsh sends a message to the screen immediately whenever a background job completes. Ordinarily tcsh notifies you about job completion just before displaying the next prompt. Refer to "Job Control" on page 272.

pushdtohome

Causes a call to pushd without any arguments to change directories to your home directory (equivalent to pushd ).

pushdsilent

Causes pushd and popd not to display the directory stack.

rmstar

Causes the shell to request confirmation when you give an rm * command.

verbose

Causes the shell to display each command after a history expansion (page 342). Set verbose by calling tcsh with the v option or by using set.

visiblebell

Causes audible beeps to be replaced by flashing the screen.


Control Structures

The TC Shell uses many of the same control structures as the Bourne Again Shell. In each case the syntax is different, but the effects are the same. This section summarizes the differences between the control structures in the two shells. For more information refer to "Control Structures" on page 524.

if

The syntax of the if control structure is

if (expression) simple-command

The if control structure works only with simple commands, not with pipes or lists of commands. You can use the if...then control structure (page 368) to execute more complex commands.

tcsh $ cat if_1
#!/bin/tcsh
# Routine to show the use of a simple if control structure.
#
if ( $#argv == 0 ) echo "if_1: there are no arguments"

The if_1 script checks whether it was called without any arguments. If the expression enclosed in parentheses evaluates to truethat is, if zero arguments were on the command linethe if structure displays a message.

In addition to logical expressions such as the one the if_1 script uses, you can use expressions that return a value based on the status of a file. The syntax for this type of expression is

n filename

where n is from the list in Table 9-8.

Table 9-8. Value of n

n

Meaning

b

File is a block special file

c

File is a character special file

d

File is a directory file

e

File exists

f

File is an ordinary or directory file

g

File has the set-group-ID bit set

k

File has the sticky bit (page 955) set

l

File is a symbolic link

o

File is owned by user

p

File is a named pipe (FIFO)

r

The user has read access to the file

s

File is not empty (has nonzero size)

S

File is a socket special file

t

File descriptor (a single digit replacing filename) is open and connected to the screen

u

File has the set-user-ID bit set

w

User has write access to the file

x

User has execute access to the file

X

File is either a builtin or an executable found by searching the directories in $path

z

File is 0 bytes long

If the result of the test is true, the expression has a value of 1; if it is false, the expression has a value of 0. If the specified file does not exist or is not accessible, tcsh evaluates the expression as 0. The following example checks whether the file specified on the command line is an ordinary or directory file (and not a device or other special file):

tcsh $ cat if_2
#!/bin/tcsh
if -f $1 echo "Ordinary or Directory file"

You can combine operators where it makes sense. For example, ox filename is true if you own and have execute permission for the file. This expression is equivalent to o filename && x filename.

Some operators return useful information about a file other than reporting true or false. They use the same n filename format, where n is one of the values shown in Table 9-9.

Table 9-9. Value of n

n

Meaning

A

The last time the file was accessed.[*]

A:

The last time the file was accessed displayed in a human-readable format.

M

The last time the file was modified.[*]

M:

The last time the file was modified displayed in a human-readable format.

C

The last time the file's inode was modified.[*]

C:

The last time the file's inode was modified displayed in a human-readable format.

D

Device number for the file. This number uniquely identifies the device (disk partition, for example) on which the file resides.

I

Inode number for the file. The inode number uniquely identifies a file on a particular device.

F

A string of the form device:inode. This string uniquely identifies a file anywhere on the system.

N

Number of hard links to the file.

P

The file's permissions, shown in octal, without a leading 0.

U

Numeric user ID of the file's owner.

U:

Username of the file's owner.

G

Numeric group ID of the file's group.

G:

Name of the file's group.

Z

Number of bytes in the file.

[*] Time measured in seconds from the epoch (usually the start of January 1, 1970).

You can use only one of these operators in a given test, and it must appear as the last operator in a multiple-operator sequence. Because 0 can be a valid response from some of these operators (for instance, the number of bytes in a file might be 0), most return 1 on failure instead of the 0 that the logical operators return on failure. The one exception is F, which returns a colon if it cannot determine the device and inode for the file.

When you want to use one of these operators outside of a control structure expression, you can use the filetest builtin to evaluate a file test and report the result:

tcsh $ filetest -z if_1
0
tcsh $ filetest -F if_1
2051:12694
tcsh $ filetest -Z if_1
131

goto

The goto statement has the following syntax:

goto label

A goto builtin transfers control to the statement beginning with label:. The following script fragment demonstrates the use of goto:

tcsh $ cat goto_1
#!/bin/tcsh
#
# test for 2 arguments
#
if ($#argv == 2) goto goodargs
echo "Usage: goto_1 arg1 arg2"
exit 1
goodargs:
...

The goto_1 script displays a usage message (page 528) when it is called with more or fewer than two arguments.

Interrupt Handling

The onintr statement transfers control when you interrupt a shell script. The format of an onintr statement is

onintr label

When you press the interrupt key during execution of a shell script, the shell transfers control to the statement beginning with label:. This statement allows you to terminate a script gracefully when it is interrupted. You can use it to ensure that when you interrupt a shell script, the script removes temporary files before returning control to the parent shell.

The following script demonstrates onintr. It loops continuously until you press the interrupt key, at which time it displays a message and returns control to the shell:

tcsh $ cat onintr_1
#!/bin/tcsh
# demonstration of onintr
onintr close
while ( 1 )
    echo "Program is running."
    sleep 2
end
close:
echo "End of program."

If a script creates temporary files, you can use onintr to remove them.

close:
rm -f /tmp/$$*

The ambiguous file reference /tmp/$$* matches all files in /tmp that begin with the PID number of the current shell. Refer to page 562 for a description of this technique for naming temporary files.

if...then...else

The if...then...else control structure has three forms. The first form, an extension of the simple if structure, executes more complex commands or a series of commands if expression is true. This form is still a one-way branch.

if (expression) then
  commands
endif

The second form is a two-way branch. If expression is true, the first set of commands is executed. If it is false, the set of commands following else is executed.

if (expression) then
  commands
else
commands
endif

The third form is similar to the if...then...elif structure (page 531). It performs tests until it finds an expression that is true and then executes the corresponding commands.

if (expression) then
     commands
else if (expression) then
   commands
. . .
else
commands
endif

The following program assigns a value of 0, 1, 2, or 3 to the variable class based on the value of the first command line argument. The program declares the variable class at the beginning for clarity; you do not need to declare it before its first use. Also for clarity, the script assigns the value of the first command line argument to number.

tcsh $ cat if_else_1
#!/bin/tcsh
# routine to categorize the first
# command line argument
set class
set number = $argv[1]
#
if ($number < 0) then
    @ class = 0
else if (0 <= $number && $number < 100) then
    @ class = 1
else if (100 <= $number && $number < 200) then
    @ class = 2
else
    @ class = 3
endif
#
echo "The number $number is in class ${class}."

The first if statement tests whether number is less than 0. If it is, the script assigns 0 to class and transfers control to the statement following endif. If it is not, the second if tests whether the number is between 0 and 100. The && is the Boolean AND operator, yielding a value of true if the expression on each side is true. If the number is between 0 and 100, 1 is assigned to class and control is transferred to the statement following endif. A similar test determines whether the number is between 100 and 200. If it is not, the final else assigns 3 to class. The endif closes the if control structure. The final statement uses braces ({}) to isolate the variable class from the following period. The braces isolate the period for clarity; the shell does not consider a punctuation mark to be part of a variable name. The braces would be required if you wanted other characters to follow immediately after the variable.

foreach

The foreach structure parallels the bash for...in structure (page 537). The syntax is

foreach loop-index (argument-list)
    commands
end

This structure loops through commands. The first time through the loop, the structure assigns the value of the first argument in argument-list to loop-index. When control reaches the end statement, the shell assigns the value of the next argument from argument-list to loop-index and executes the commands again. The shell repeats this procedure until it exhausts argument-list.

The following tcsh script uses a foreach structure to loop through the files in the working directory containing a specified string of characters in their filename and to change the string. For example, you can use it to change the string memo in filenames to letter. The filenames memo.1, dailymemo, and memories would change to letter.1, dailyletter, and letterries.

This script requires two arguments: the string to be changed (the old string) and the new string. The argument-list of the foreach structure uses an ambiguous file reference to loop through all filenames that contain the first argument. For each filename that matches the ambiguous file reference, the mv utility changes the filename. The echo and sed commands appear within back ticks (`) that indicate command substitution: Executing the commands within the back ticks replaces the back ticks and everything between them. Refer to "Command Substitution" on page 327 for more information. The sed utility (page 641) substitutes the first argument for the second argument in the filename. The $1 and $2 are abbreviated forms of $argv[1] and $argv[2].

tcsh $ cat ren
#!/bin/tcsh
# Usage:        ren arg1 arg2
#               changes the string arg1 in the names of files
#               in the working directory into the string arg2
if ($#argv != 2) goto usage
foreach i ( *$1* )
   mv $i `echo $i  sed -n s/$1/$2/p`
end
exit 0
usage:
echo "Usage: ren arg1 arg2"
exit 1

Optional

The next script uses a foreach loop to assign the command line arguments to the elements of an array named buffer:

tcsh $ cat foreach_1
#!/bin/tcsh
# routine to zero-fill argv to 20 arguments
#
set buffer = (0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)
set count = 1
#
if ($#argv > 20) goto toomany
#
foreach argument ($argv[*])
    set buffer[$count] = $argument
    @ count++
end
# REPLACE command ON THE NEXT LINE WITH
# THE PROGRAM YOU WANT TO CALL.
exec command $buffer[*]
#
toomany:
echo "Too many arguments given."
echo "Usage: foreach_1 [up to 20 arguments]"
exit 1

The foreach_1 script calls another program named command with a command line guaranteed to contain 20 arguments. If foreach_1 is called with fewer than 20 arguments, it fills the command line with zeros to complete the 20 arguments for command. Providing more than 20 arguments causes it to display a usage message and exit with an error status of 1.

The foreach structure loops through the commands one time for each command line argument. Each time through the loop, foreach assigns the value of the next argument from the command line to the variable argument. Then the script assigns each of these values to an element of the array buffer. The variable count maintains the index for the buffer array. A postfix operator increments the count variable using @ (@ count++). The exec builtin (bash and tcsh; page 574) calls command so that a new process is not initiated. (Once command is called, the process running this routine is no longer needed so a new process is not required.)

while

The syntax of the while structure is

while (expression)
   commands
end

This structure continues to loop through commands while expression is true. If expression is false the first time it is evaluated, the structure never executes commands.

tcsh $ cat while_1
#!/bin/tcsh
# Demonstration of a while control structure.
# This routine sums the numbers between 1 and n,
# with n being the first argument on the command # line.
#
set limit = $argv[1]
set index = 1
set sum = 0
#
while ($index <= $limit)
    @ sum += $index
    @ index++
end
#
echo "The sum is $sum"

This program computes the sum of all integers up to and including n, where n is the first argument on the command line. The += operator assigns the value of sum + index to sum.

break and continue

You can interrupt a foreach or while structure with a break or continue statement. These statements execute the remaining commands on the line before they transfer control. The break statement transfers control to the statement after the end statement, terminating execution of the loop. The continue statement transfers control to the end statement, which continues execution of the loop.

switch

The switch structure is analogous to the bash case structure (page 545):

switch (test-string)
  case pattern:
commands
breaksw
case pattern:
commands
breaksw
...
default:
  commands
breaksw
endsw

The breaksw statement transfers control to the statement following the endsw statement. If you omit a breaksw, control falls through to the next command. You can use any of the special characters listed in Table 13-2 on page 546 within pattern except the pipe symbol (|).

tcsh $ cat switch_1
#!/bin/tcsh
# Demonstration of a switch control structure.
# This routine tests the first command line argument
# for yes or no in any combination of uppercase and
# lowercase letters.
#
# test that argv[1] exists
if ($#argv != 1) then
    echo "Usage: switch_1 [yes|no]"
    exit 1
else
# argv[1] exists, set up switch based on its value
   switch ($argv[1])
   # case of YES
         case [yY][eE][sS]:
         echo "Argument one is yes."
         breaksw
    #
    # case of NO
         case [nN][o0]:
         echo "Argument one is no."
    breaksw
    #
    # default case
         default:
         echo "Argument one is neither yes nor no."
         breaksw
    endsw
endif


Builtins

Builtins are commands that are part of (built into) the shell. When you give a simple filename as a command, the shell first checks whether it is the name of a builtin. If it is, the shell executes it as part of the calling process; the shell does not fork a new process to execute the builtin. The shell does not need to search the directory structure for builtin programs because they are immediately available to the shell.

If the simple filename you give as a command is not a builtin, the shell searches the directory structure for the program you want, using the PATH variable as a guide. When it finds the program the shell forks a new process to execute the program.

Although they are not listed in Table 9-10, the control structure keywords (if, foreach, endsw, and so on) are builtins. The table describes many of the tcsh builtins, some of which are also built into other shells.

Table 9-10. tcsh builtins

Builtin

Function

% job

A synonym for the fg builtin. The job is the job number of the job you want to bring to the foreground (page 273).

% job &

A synonym for the bg builtin. The job is the number of the job you want to put in the background (page 274).

@

Similar to the set builtin but evaluates numeric expressions. Refer to "Numeric Variables" on page 355.

alias

Creates and displays aliases; bash uses a different syntax than tcsh. Refer to "Aliases" on page 344.

alloc

Displays a report of the amount of free and used memory.

bg

Moves a suspended job into the background (page 274).

bindkey

Controls the mapping of keys to the tcsh command line editor commands.

bindkey

Without any arguments, bindkey lists all key bindings (page 350).

bindkeyl

Lists all available editor commands along with a short description of each.

bindkeye

Puts the command line editor in emacs mode (page 350).

bindkeyv

Puts the command line editor in vi(m) mode (page 350).

bindkey key command

Attaches the editor command command to the key key.

bindkeyb key command

Similar to the previous form but allows you to specify control keys by using the form Cx (where x is the character you type while you press the CONTROL key), specify meta key sequences as Mx (on most keyboards used with Mac OS X, the OPTION key is the meta key, but you have to set an option in the Terminal utility to use it; see page 31), and specify function keys as F-x.

bindkeyc key command

Binds the key key to the command command. Here the command is not an editor command but either a shell builtin or an executable program.

bindkeys key string

Whenever you type key, string is substituted.

builtins

Displays a list of all builtins.

cd or chdir

Changes working directories (page 82).

dirs

Displays the directory stack (page 275).

echo

Displays its arguments. You can prevent echo from displaying a RETURN at the end of a line by using the n option (see "Reading User Input" on page 358) or by using a trailing \c (see "read: Accepts User Input" on page 571). The echo builtin is similar to the echo utility (page 720).

eval

Scans and evaluates the command line. When you put eval in front of a command, the command is scanned twice by the shell before it is executed. This feature is useful with a command that is generated by command or variable substitution. Because of the order in which the shell processes a command line, it is sometimes necessary to repeat the scan to achieve the desired result. See an example of the use of eval on page 316.

exec

Overlays the program currently being executed with another program in the same shell. The original program is lost. Refer to "exec: Executes a Command" on page 574 for more information; also refer to source (page 376).

exit

Exits from a TC Shell. When you follow it with a numeric argument, tcsh returns that number as the exit status (page 564).

fg

Moves a job into the foreground (page 272).

filetest

Takes one of the file inquiry operators followed by one or more filenames and applies the operator to each filename (page 367). Returns the results as a space-separated list.

glob

Like echo, but does not display SPACEs between its arguments and does not follow its display with a NEWLINE.

hashstat

Reports on the efficiency of tcsh's hash mechanism. The hash mechanism speeds the process of searching through the directories in your search path. See also rehash (page 376) and unhash (page 377).

history

Displays a list of recent commands (page 342).

jobs

Displays a list of jobs (suspended commands and those running in the background).

kill

Terminates a job or process (page 580).

limit

Limits the computer resources that the current process and any processes it creates can use. You can put limits on the number of seconds of CPU time the process can use, the size of files that the process can create, and so forth.

log

Immediately produces the report that the watch shell variable (page 362) would normally produce every 10 minutes.

login

Logs in a user. Can be followed by a username.

logout

Ends a session if you are using your original (login) shell.

lsF

Similar to lsF but faster. (This builtin is the characters lsF without any SPACEs.)

nice

Lowers the processing priority of a command or a shell. It is useful if you want to run a command that makes large demands on the system and you do not need the output right away. If you are Superuser, you can use nice to raise the priority of a command. Refer to page 794 for more information on the nice builtin and the nice utility, which is available from bash.

nohup

Allows you to log out without terminating processes running in the background. Some systems are set up to do this automatically. Refer to page 798 for information on the nohup builtin and the nohup utility, which is available from bash.

notify

Causes the shell to notify you immediately when the status of one of your jobs changes (page 272).

onintr

Controls what action an interrupt causes within a script (page 367). See "trap: Catches a Signal" on page 577 for information on the equivalent command in bash.

popd

Removes a directory from the directory stack (page 275).

printenv

Displays all environment variable names and values.

pushd

Changes the working directory and places the new directory at the top of the directory stack (page 275).

rehash

Re-creates the internal tables used by the hash mechanism. Whenever a new instance of tcsh is invoked, the hash mechanism creates a sorted list of all available commands based on the value of path. After you add a command to a directory in path, use rehash to re-create the sorted list of commands. If you do not, tcsh may not be able to find the new command. Also refer to hashstat (page 375) and unhash (page 377).

repeat

Takes two argumentsa count and simple command (no pipes or lists of commands)and repeats the command the number of times specified by the count.

sched

Executes a command at a specified time. For example, the following command causes the shell to print the message Dental appointment. at 10 AM:

tcsh $ sched 10:00 echo "Dental appointment."

Without any arguments, sched prints the list of scheduled commands. When the time to execute a scheduled command arrives, tcsh executes the command just before it displays a prompt.

set

Declares, initializes, and displays local variables (page 352).

setenv

Declares, initializes, and displays environment variables (page 352).

shift

Analogous to the bash shift builtin (page 567). Without an argument, shift promotes the indexes of the argv array. You can use it with an argument of an array name to perform the same operation on that array.

source

Executes the shell script given as its argument: source does not fork another process. It is similar to the bash. (dot) builtin (page 261). The source builtin expects a TC Shell script so no leading #! is required in the script. The current shell executes source so that the script can contain commands, such as set, that affect the current shell. After you make changes to your .tcshrc or .login file, you can use source to execute it from the shell and thereby put the changes into effect without logging off and on. You can nest source builtins.

stop

Stops a job or process that is running in the background. The stop builtin accepts multiple arguments.

suspend

Stops the current shell and puts it in the background. It is similar to the suspend key, which stops jobs running in the foreground.

time

Executes the command that you give it as an argument. It displays a summary of time-related information about the executed command, according to the time shell variable (page 361). Without an argument, time displays the times for the current shell and its children.

umask

Identifies or changes the access permissions that are assigned to files you create (page 883).

unalias

Removes an alias (page 344).

unhash

Turns off the hash mechanism. See also hashstat (page 375) and rehash (page 376).

unlimit

Removes limits (page 375) on the current process.

unset

Removes a variable declaration (page 352).

unsetenv

Removes an environment variable declaration (page 352).

wait

Causes the shell to wait for all child processes to terminate. When you give a wait command in response to a shell prompt, tcsh does not display a prompt until all background processes have finished execution. If you interrupt it with the interrupt key, wait displays a list of outstanding processes before tcsh displays a prompt.

where

When given the name of a command as an argument, locates all occurrences of the command and, for each, tells you whether it is an alias, a builtin, or an executable program in your path.

which

Similar to where but reports on only the command that would be executed, not all occurrences. This builtin is much faster than the Mac OS X which utility and knows about aliases and builtins.


Chapter Summary

Like the Bourne Again Shell, the TC Shell is both a command interpreter and a programming language. The TC Shell, which is based on the C Shell that was developed at the University of California at Berkeley, includes popular features such as history, alias, and job control.

You may prefer to use tcsh as a command interpreter, especially if you are familiar with the C Shell. You can use chsh to change your login shell to tcsh. However, running tcsh as your interactive shell does not cause tcsh to run shell scripts; they will continue to be run by bash unless you explicitly specify another shell on the first line of the script or specify the script name as an argument to tcsh. Specifying the shell on the first line of a shell script ensures the behavior you expect.

If you are familiar with bash, you will notice some differences between the two shells. For instance, the syntax you use to assign a value to a variable differs and tcsh allows SPACEs around the equal sign. Both numeric and nonnumeric variables are created and given values using the set builtin. The @ builtin can evaluate numeric expressions for assignment to numeric variables.

setenv

Because there is no export builtin in tcsh, you must use the setenv builtin to create an environment (global) variable. You can also assign a value to the variable with the setenv command. The command unset removes both local and environment variables, whereas the command unsetenv removes only environment variables.

Aliases

The syntax of the tcsh alias builtin is slightly different from that of alias in bash. Unlike bash, the tcsh aliases permit you to substitute command line arguments using the history mechanism syntax.

Most other tcsh features, such as history, word completion, and command line editing, closely resemble their bash counterparts. The syntax of the tcsh control structures is slightly different but provides functionality equivalent to that found in bash.

Globbing

The term globbing, a carryover from the original Bourne Shell, refers to the matching of strings containing special characters (such as * and ?) to filenames. If tcsh is unable to generate a list of filenames matching a globbing pattern, it displays an error message. This behavior contrasts with that of bash, which simply leaves the pattern alone.

Standard input and standard output can be redirected in tcsh, but there is no straightforward way to redirect them independently. Doing so requires the creation of a subshell that redirects standard output to a file while making standard error available to the parent process.


Exercises

1.

Assume that you are working with the following history list:

37 mail alex
38 cd /Users/jenny/correspondence/business/cheese_co
39 less letter.0321
40 vim letter.0321
41 cp letter.0321 letter.0325
42 grep hansen letter.0325
43 vim letter.0325
44 lpr letter*
45 cd ../milk_co
46 pwd
47 vim wilson.0321 wilson.0329

Using the history mechanism, give commands to

  1. Send mail to Alex.

  2. Use vim to edit a file named wilson.0329.

  3. Send wilson.0329 to the printer.

  4. Send both wilson.0321 and wilson.0329 to the printer.

2.

How can you display the aliases currently in effect? Write an alias named homedots that lists the names (only) of all invisible files in your home directory.

3.

How can you prevent a command from sending output to the terminal when you start it in the background? What can you do if you start a command in the foreground and later decide that you want it to run in the background?

4.

What statement can you put in your ~/.tcshrc file to prevent accidentally overwriting a file when you redirect output? How can you override this feature?

5.

Assume that the working directory contains the following files:

adams.ltr.03
adams.brief
adams.ltr.07
abelson.09
abelson.brief
anthony.073
anthony.brief
azevedo.99

What happens if you press TAB after typing the following commands?

  1. less adams.1

  2. cat a

  3. ls ant

  4. file az

    What happens if you press CONTROL-D after typing the following commands?

  5. ls ab

  6. less a

6.

Write an alias named backup that takes a filename as an argument and creates a copy of that file with the same name and a filename extension of .bak.

7.

Write an alias named qmake (quiet make) that runs make with both standard output and standard error redirected to the file named make.log. The command qmake should accept the same options and arguments as make.

8.

How can you make tcsh always display the pathname of the working directory as part of its prompt?


Advanced Exercises

9.

What lines do you need to change in the Bourne Again Shell script command_menu (page 547) to turn it into a TC Shell script? Make the changes and verify that the new script works.

10.

Users often find rm (and even rmi) too unforgiving because it removes files irrevocably. Create an alias named delete that moves files specified by its argument(s) into the ~/.trash directory. Create a second alias named undelete that moves a file from the ~/.trash directory into the working directory. Put the following line in your ~/.logout file to remove any files that you deleted during the login session:

/bin/rm -f $HOME/.trash/* >& /dev/null

Explain what could be different if the following line were put in your ~/.logout file instead:

rm $HOME/.trash/*

11.

Modify the foreach_1 script (page 370) so that it takes the command to exec as an argument.

12.

Rewrite the program while_1 (page 371) so that it runs faster. Use the time builtin to verify the improvement in execution time.

13.

Write your own version of find named myfind that writes output to the file findout but without the clutter of error messages, such as those generated when you do not have permission to search a directory. The myfind command should accept the same options and arguments as find. Can you think of a situation in which myfind does not work as desired?

14.

When the foreach_1 script (page 370) is supplied with 20 or fewer arguments, why are the commands following toomany: not executed? (Why is there no exit command?)


Part IV: Networking and Maintenance

CHAPTER 10 NETWORKING AND THE INTERNET

CHAPTER 11 SYSTEM MAINTENANCE


Chapter 10. Networking and the Internet

Previous Page
Next Page

10. Networking and the Internet

IN THIS CHAPTER

Types of Networks and How They Work | 385
Network Protocols | 391
Network Utilities | 403
ping: Tests a Network Connection | 405
traceroute: Traces a Route over the Internet | 406
host and dig: Query Internet Nameservers | 408
Distributed Computing | 410
Usenet | 418
WWW: World Wide Web | 420

The communications facilities linking computers are continually improving, allowing faster and more economical connections. The earliest computers were unconnected stand-alone systems. To transfer information from one system to another, you had to store it in some form (usually magnetic tape, paper tape, or punch cardscalled IBM or Hollerith cards), carry it to a compatible system, and read it back in. A notable advance occurred when computers began to exchange data over serial lines, although the transfer rate was slow (hundreds of bits per second). People quickly invented new ways to take advantage of this computing power, such as email, news retrieval, and bulletin board services. With the speed of today's networks, a piece of email can cross the country or even travel halfway around the world in a few seconds.

Today it would be difficult to find a computer facility that does not include a LAN to link its systems. Mac OS X systems are typically attached to an Ethernet (page 931) network. Wireless networks are also prevalent, especially Apple's Airport and Airport Extreme. Large computer facilities usually maintain several networks, often of different types, and almost certainly have connections to larger networks (companywide or campuswide and beyond).

Internet

The Internet is a loosely administered network of networks (an internetwork) that links computers on diverse LANs around the globe. An internet (small i) is a generic network of networks that may share some parts in common with the public Internet. It is the Internet that makes it possible to send an email message to a colleague thousands of miles away and receive a reply within minutes. A related term, intranet, refers to the networking infrastructure within a company or other institution. Intranets are usually private; access to them from external networks may be limited and carefully controlled, typically using firewalls (page 389).

Network services

Over the past decade many network services have emerged and become standardized. On Mac OS X and other UNIX and UNIX-like systems, special processes called daemons (page 929) support such services by exchanging specialized messages with other systems over the network. Several software systems have been created to allow computers to share filesystems with one another, making it appear as though remote files are stored on local disks. Sharing remote filesystems allows users to share information without knowing where the files physically reside, without making unnecessary copies, and without learning a new set of utilities to manipulate them. Because the files appear to be stored locally, you can use standard utilities (such as cat, vim, lpr, mv, or their graphical counterparts) to work with them.

Developers have been creating new tools and extending existing ones to take advantage of higher network speeds and to work within more crowded networks. The rlogin, rsh, and telnet utilities, which were designed long ago, have largely been supplanted by ssh (secure shell, page 847) in recent years. The ssh utility allows a user to log in on or execute commands securely on a remote computer. Users rely on such utilities as scp and ftp to transfer files from one system to another across the network. Communication utilities, including email utilities and chat programs (e.g., talk, Internet Relay Chat [IRC], ICQ, and instant messenger [IM] programs, such as iChat, AOL's AIM, and gaim) have become so prevalent that many people with very little computer expertise use them on a daily basis to keep in touch with friends, family, and colleagues.

Intranet

An intranet is a network that connects computing resources at a school, company, or other organization but, unlike the Internet, typically restricts access to internal users. An intranet is very similar to a LAN (local area network) but is based on Internet technology. An intranet can provide database, email, and Web page access to a limited group of people, regardless of their geographic location.

The ability of an intranet to connect dissimilar machines is one of its strengths. Think of all the machines that are on the Internet: Macs, PCs running different versions of Windows, various machines running UNIX and Linux, and so on. Each of these machines can communicate via IP (page 391), a common protocol. So it is with an intranet: Different machines can all talk to one another.

Another key difference between the Internet and an intranet is that the Internet transmits only one protocol suite: IP. An intranet can be set up to use a number of protocols, such as IP, IPX, AppleTalk, DECnet, XNS, or various other protocols developed by vendors over the years. Although these protocols cannot be transmitted directly over the Internet, you can set up special gateway boxes at remote sites that tunnel or encapsulate these protocols into IP packets and then use the Internet to pass them.

You can use an extranet (also called a partner net) or a virtual private network (VPN) to improve security. These terms describe ways to connect remote sites securely to a local site, typically by using the public Internet as a carrier and using encryption as a means of protecting data in transit.

As with the Internet, the communications potential of intranets is boundless. You can set up a private chat between people at remote locations, access a company database, see what is new at school, or read about the new university president. Companies that developed products for use on the Internet are investing increasingly more time and money developing intranet software applications as the intranet market explodes.

Following are some terms you may want to become familiar with before you read the rest of this chapter. Refer to the Glossary on page 919 for their definitions.

ASP | firewall | internet | ISP | sneakernet
bridge | gateway | Internet | packet | switch
extranet | hub | intranet | router | VPN


Types of Networks and How They Work

Computers communicate over networks using unique addresses assigned by system software. A computer message, called a packet, frame, or datagram, includes the address of the destination computer and the sender's return address. The three most common types of networks are broadcast, point-to-point, and switched. Once popular token-based networks (such as FDDI and token ring) are rarely seen anymore.

Speed is important to the proper functioning of the Internet. Newer specifications (cat 6 and cat 7) are being standardized for 1000BaseT (1 gigabit per second, called gigabit Ethernet, or GIG-E) and faster networking. Some of the networks that form the backbone of the Internet run at speeds up to almost 10 gigabits per second (OC192) to accommodate the ever-increasing demand for network services. Table 10-1 lists some of the specifications in use today.

Table 10-1. Network specifications

Specification

Speed

DS0

64 kilobits per second

ISDN

Two DS0 lines plus signaling (16 kilobits per second) or 128 kilobits per second

T-1

1.544 megabits per second (24 DS0 lines)

T-3

43.232 megabits per second (28 T-1s)

OC3

155 megabits per second (100 T-1s)

OC12

622 megabits per second (4 OC3s)

OC48

2.5 gigabits per seconds (4 OC12s)

OC192

9.6 gigabits per second (4 OC48s)

Broadcast Networks

On a broadcast network, such as Ethernet, any of the many systems attached to the network cable can send a message at any time; each system examines the address in each message and responds only to messages addressed to it. A problem occurs on a broadcast network when multiple systems send data at the same time, resulting in a collision of the messages on the cable. When messages collide, they can become garbled. The sending system notices the garbled message and resends it after waiting a short but random amount of time. Waiting a random amount of time helps prevent those same systems from resending the data at the same moment and experiencing yet another collision. The extra traffic that results from collisions can put an extra load on the network; if the collision rate gets too high, retransmissions result in more collisions. Ultimately the network becomes unusable.

Point-to-Point Networks

A point-to-point link does not seem like much of a network because only two endpoints are involved. However, most connections to WANs (wide area networks) are through point-to-point links, using wire cable, radio, or satellite links. The advantage of a point-to-point link is its simplicity: Because only two systems are involved, the traffic on the link is limited and well understood. A disadvantage is that each system can typically be equipped for a small number of such links, and it is impractical and costly to establish point-to-point links that connect each computer to all the rest.

Point-to-point links often use serial lines and modems. The combination of a modem with a point-to-point link allows an isolated system to connect inexpensively to a larger network.

The most common types of point-to-point links are the ones used to connect to the Internet. When you use DSL[1] (digital subscriber line), you are using a point-to-point link to connect to the Internet. Serial lines, such as T-1, T-3, ATM links, and ISDN, are all point-to-point. Although it might seem like a point-to-point link, a cable modem is based on broadcast technology and in that way is similar to Ethernet.

[1] The term DSL incorporates the xDSL suite of technologies, including ADSL, XDSL, SDSL, and HDSL.

Switched Networks

A switch is a device that establishes a virtual path between source and destination hosts in such a way that each path appears to be a point-to-point link, much like a railroad roundhouse. The switch creates and tears down virtual paths as hosts need to communicate with each other. Each host thinks it has a direct point-to-point path to the host it is talking to. Contrast this approach with a broadcast network, where each host also sees traffic bound for other hosts. The advantage of a switched network over a pure point-to-point network is that each host requires only one connection: the connection to the switch. Using pure point-to-point connections, each host must have a connection to every other host. Scalability is provided by further linking switches.

LAN: Local Area Network

Local area networks (LANs) are confined to a relatively small areaa single computer facility, building, or campus. Today, most LANs run over copper or fiberoptic (glass or plastic) cable, but other wireless technologies, such as infrared (similar to most television remote control devices) and radio wave (wireless, or Wi-Fi), are becoming more popular.

If its destination address is not on the local network, a packet must be passed on to another network by a router (page 388). A router may be a general-purpose computer or a special-purpose device attached to multiple networks to act as a gateway among them.

Ethernet

A Mac OS X system connected to a LAN usually connects to a network using Ethernet. A typical Ethernet connection can support data transfer rates from 10 megabits per second to 1 gigabit per second, with speed enhancements planned for the future. As a result of computer load, competing network traffic, and network overhead, file transfer rates on an Ethernet are always slower than the maximum, theoretical transfer rate.

Cables

An Ethernet network transfers data using copper or fiberoptic cable or wireless transmitters and receivers. Originally each computer was attached to a thick coaxial cable (called thicknet) at tap points spaced at six-foot intervals along the cable. The thick cable was awkward to deal with, so other solutions, including a thinner coaxial cable called thinnet, or 10Base2,[2] were developed. Today most Ethernet connections either are wireless or are made over unshielded twisted pair (referred to as UTP, Category 5 [cat 5], Category 5e [cat 5e], Category 6 [cat 6], 10BaseT, or 100BaseT) wiresimilar to the type of wire used for telephone lines and serial data communications.

[2] Versions of Ethernet are classified as XBaseY, where X is the data rate in megabits per second, Base means baseband (as opposed to radio frequency), and Y is the category of cabling.

Switch

A switched Ethernet network is a special case of a broadcast network that works with a network switch (or just switch) which is a special class of hub that has intelligence. Instead of having a dumb repeater (passive hub) that broadcasts every packet it receives out of every port, a switch learns which devices are connected to which of its ports. A switch sorts packets and then sends the traffic to only the machine it is intended for. A switch also has buffers for holding and queuing packets.

Some Ethernet switches have enough bandwidth to communicate simultaneously, in full-duplex mode, with all the devices connected to them. A nonswitched (hub-based) broadcast network can run in only half-duplex mode. Full-duplex Ethernet further improves things by eliminating collisions. Each host on a switched network can transmit and receive simultaneously at 10/100/1,000 megabits per second for an effective bandwidth between hosts of 20/200/2,000 megabits per second, depending on the capacity of the switch.

Wireless

Wireless networks are becoming increasingly common. They are found in offices, homes, and public places, such as universities, coffee shops, and airports. Wireless access points provide functionality similar to an Ethernet hub. They allow multiple users to interact via a common radio frequency spectrum. A wireless, point-to-point connection allows you to wander about your home or office with a laptop, using an antenna to link to a LAN or to the Internet via an in-house base station. Mac OS X is designed to work with Apple wireless boards, although third-party wireless boards exist and provide drivers. A wireless access point, or base station, connects a wireless network to a wired network so that no special protocol is required for a wireless connection. You can set up a Macintosh system to serve as an access point using Apple Airport software.

WAN: Wide Area Network

A wide area network (WAN) covers a large geographic area. In contrast, the technologies (such as Ethernet) used for LANs were designed to work over limited distances and for a certain number of host connections. A WAN may span long distances over dedicated data lines (leased from a telephone company) or radio or satellite links. WANs are often used to interconnect LANs. Major Internet service providers rely on WANs to connect to their customers within a country and around the globe.

MAN

Some networks do not fit into either the LAN or the WAN designation. A MAN (metropolitan area network) is a network that is contained in a smaller geographic area, such as a city. Like WANs, MANs are typically used to interconnect LANs.

Internetworking Through Gateways and Routers

Gateway

A LAN connects to a WAN through a gateway, a generic term for a computer or a special device with multiple network connections that passes data from one network to another. A gateway converts the data traffic from the format used on the LAN to that used on the WAN. Data that crosses the country from one Ethernet to another over a WAN, for example, is repackaged from the Ethernet format to a different format that can be processed by the communications equipment that makes up the WAN backbone. When it reaches the end of its journey over the WAN, the data is converted by another gateway to a format appropriate for the receiving network. For the most part, these details are of concern only to the network administrators; the end user does not need to know anything about how the data transfer takes place.

Router

A router is the most popular form of gateway. Routers play an important role in internetworking. Just as you might study a map to plan your route when you need to drive to an unfamiliar place, a computer needs to know how to deliver a message to a system attached to a distant network by passing through intermediary systems and networks along the way. You might envision using a giant network road map to choose the route that your data should follow, but a static map of computer routes is usually a poor choice for a large network. Computers and networks along the route you choose may be overloaded or down, without providing a detour for your message.

Routers communicate with one another dynamically, keeping each other informed about which routes are open for use. To extend the analogy, this situation would be like heading out on a car trip without consulting a map to find a route to your destination; instead you head for a nearby gas station and ask directions. Throughout the journey, you would continue to stop at one gas station after another, getting directions at each to find the next one. Although it would take a while to make the stops, each gas station would advise you of bad traffic, closed roads, alternative routes, and shortcuts.

The stops made by the data are much quicker than those you would make in your car, but each message leaves each router on a path chosen based on the most current information. Think of it as a GPS (global positioning system) setup that automatically gets updates at each intersection and tells you where to go next, based on traffic and highway conditions.

Figure 10-1 (next page) shows an example of how LANs might be set up at three sites interconnected by a WAN (the Internet). In this type of network diagram, Ethernet LANs are drawn as straight lines, with devices attached at right angles; WANs are represented as clouds, indicating that the details have been left out; wireless connections are drawn as zigzag lines with breaks, indicating that the connection may be intermittent.

Figure 10-1. A slice of the Internet

[View full size image]

In Figure 10-1, a gateway or a router relays messages between each LAN and the Internet. Three of the routers in the Internet are shown (for example, the one closest to each site). Site A has a server, a workstation, a network computer, and a PC sharing a single Ethernet LAN. Site B has an Ethernet LAN that serves a printer and four workstations. A firewall permits only certain traffic to pass between the Internet router and the site's local router. Site C has three LANs linked by a single router, perhaps to reduce the traffic load that would result if they were combined or to keep workgroups or locations on separate networks. Site C also includes a wireless access point that enables wireless communication with nearby computers.

Firewall

A firewall in a car separates the engine compartment from the passenger compartment, protecting the driver and passengers from engine fires, noise, and fumes. In much the same way, computer firewalls separate computers from malicious and unwanted users.

A firewall prevents certain types of traffic from entering or leaving a network. For example, a firewall might prevent traffic from your IP address from leaving the network and prevent anyone except users from selected domains from using FTP to retrieve data from the network. The implementations of firewalls vary widely, from UNIX machines with two interfaces (page 937) running custom software to a router (page 952) with simple access lists to esoteric, vendor supplied firewall appliances. Most larger installations have at least one kind of firewall in place. A firewall is often accompanied by a proxy server/gateway (page 416) that provides an intermediate point between you and the host you are communicating with.

In addition to the firewalls found in multipurpose computers, firewalls are becoming increasingly common in consumer appliances. For example, they are built into cable modems, wireless gateways, routers, and stand-alone devices.

Typically a Mac OS X system includes a minimal firewall. A small group of Mac OS X systems may have an inexpensive Linux or UNIX system with two network interfaces and packet-filtering software functioning as a dedicated firewall. One of the interfaces connects to the Internet, modems, and other outside data sources. The other interface connects, normally through a hub or switch, to the local network.

Network Protocols

To exchange information over a network, computers must communicate using a common language, or protocol (page 949). The protocol determines the format of message packets. The predominant network protocols used by Mac OS X systems are TCP and IP, referred to as TCP/IP[3] (Transmission Control Protocol and Internet Protocol), although many systems also use AppleTalk. Network services that need highly reliable connections, such as ssh and scp, tend to use TCP/IP. Another protocol used for some system services is UDP (User Datagram Protocol). Network services that do not require guaranteed delivery, such as RealAudio and RealVideo, operate satisfactorily with the simpler UDP.[4]

[3] All references to IP imply IPv4 (page 938).

[4] Voice and video protocols are delay sensitive, not integrity sensitive. The human ear and eye accept and interpolate loss in an audio stream but cannot deal with variable delay. The guaranteed delivery that TCP provides introduces delay on a busy network when packets get retransmitted. This delay is not acceptable for video and audio transmissions, whereas less than 100 percent integrity is acceptable.

IP: Internet Protocol

Layering was introduced to facilitate protocol design: Layers distinguish functional differences between adjacent protocols. A grouping of layers can be standardized into a protocol model. The IP protocol has a corresponding model for what distinguishes protocol layers. The IP model differs from the ISO seven-layer protocol model (also called the OSI model) that is often illustrated in networking textbooks. Specifically IP uses the following simplified five-layer model:

  1. The first layer of the IP protocol, called the physical layer, describes the physical medium (copper, fiber, wireless) and the data encoding used to transmit signals on that medium (pulses of light, electrical waves, or radio waves, for instance).

  2. The second layer, called the data link layer, covers media access by network devices and describes how to put data into packets, transmit the data, and check it for errors. Ethernet is at this layer, as is 802.11 (page 920) wireless.

  3. The third layer, called the network layer, frequently uses IP and addresses and routes packets.

  4. The fourth layer, called the transport layer, is where TCP and UDP exist. This layer provides a means for applications to communicate with each other. Functions commonly performed by the transport layer include guaranteed delivery, delivery of packets in the order of transmission, flow control, error detection, and error correction. The transport layer is responsible for dividing data streams into packets. In addition, this layer performs port addressing, which allows it to distinguish among different services using the same transport protocol. Port addressing keeps the data from multiple applications using the same protocol (for example, TCP) separate.

  5. Anything above the transport layer is the domain of the application and is part of the fifth layer. Unlike the ISO model, the Internet model does not distinguish among application, presentation, and session layers. All the upper-layer characteristics, such as character encoding, encryption, and GUIs, are part of the application. Applications choose the transport characteristics they require as well as the corresponding transport layer protocol to send and receive data.

TCP: Transmission Control Protocol

TCP is most frequently run on top of IP in a combination referred to as TCP/IP. This protocol provides error recovery and guaranteed delivery in packet transmission order, and it works with multiple ports so that it can handle more than one application. TCP is a connection-oriented protocol (page 927), also known as a stream-based protocol. Once established, a TCP connection looks like a stream of data, not individual IP packets. The connection is assumed to remain up and be uniquely addressable. Every piece of information you write to the connection always goes to the same destination and arrives in the order it was sent. Because TCP is connection oriented and establishes a virtual circuit between two systems, this protocol is not suitable for one-to-many transmissions (see the discussion of UDP, following). TCP has builtin mechanisms for dealing with congestion (or flow) control over busy networks and throttles back (slows the speed of data flow) when it has to retransmit dropped packets. TCP can also deal with acknowledgments, wide area links, high delay links, and other situations.

UDP: User Datagram Protocol

UDP runs at layer 4 of the IP stack, just as TCP does, but is much simpler. Like TCP, UDP works with multiple ports and multiple applications. It has checksums for error detection but does not automatically retransmit datagrams (page 929) that fail the checksum test. UDP is a datagram-oriented protocol: Each datagram must carry its own address and port information. Each router along the way examines each datagram to determine the destination, one hop at a time. You can broadcast or multicast UDP datagrams to many destinations at the same time by using special addresses.

PPP: Point-to-Point Protocol

PPP provides serial line point-to-point connections that support IP. It compresses data to make the most of the limited bandwidth available on serial connections. PPP, which replaces SLIP[5] (Serial Line IP), acts as a point-to-point layer 2/3 transport that many other types of protocols can ride on. This protocol is used mostly for IP-based services and connections, such as TCP or UDP.

[5] SLIP was one of the first serial line implementations of IP and has slightly less overhead than PPP. PPP supports multiple protocols (such as AppleTalk and IPX), whereas SLIP supports only IP.

AppleTalk: A Non-IP Protocol

Mac OS X supports the AppleTalk protocol. Unlike TCP and UDP, AppleTalk is not part of the IP protocol specification, but rather is a custom protocol developed by Apple long before the introduction of OS X. AppleTalk comes in two major varieties: LocalTalk and EtherTalk. LocalTalk was used over serial ports; today it is used infrequently. EtherTalk is a way of encapsulating AppleTalk packets for transmission over Ethernet. Most routers and switches forward AppleTalk packets. If two systems can communicate via TCP/IP but not via AppleTalk, there may be a router between them that does not forward AppleTalk packets.

AppleTalk is used to share printers and files between systems. It is supported under both Mac OS X Server and the regular version of Mac OS X. Third-party support for AppleTalk is available on most BSD UNIX systems and on many Linux systems. Some versions of Windows prior to XP had builtin AppleTalk support.

Host Address

Each computer interface is identified by a unique address, or host number, on its network. A system attached to more than one network has multiple interfacesone for each network, each with a unique address.

Each packet of information that is broadcast over the network has a destination address. All hosts on the network must process each broadcast packet to see whether it is addressed to that host.[6] If the packet is addressed to a given host, that host continues to process it. If not, the host ignores it.

[6] Contrast broadcast packets with unicast packets: Ethernet hardware on a computer filters out unicast packets that are not addressed to that machine; the operating system on that machine never sees them.

The network address of a machine is an IP address, which, under IPv4, is represented as one number broken into four segments separated by periods (for example, 192.168.184.5). Domain names and IP addresses are assigned through a highly distributed system coordinated by ICANN (Internet Corporation for Assigned Names and Numbers, www.icann.org) via many registrars (see www.internic.net). ICANN is funded by the various domain name registries and registrars and by IP address registries, which supply globally unique identifiers for hosts and services on the Internet. Although you may not deal with any of these agencies directly, your Internet service provider does.

How a company uses IP addresses is determined by the system or network administrator. For example, the leftmost two sets of numbers in an IP address might represent a large network (campuswide or companywide); the third set, a subnetwork (perhaps a department or a single floor in a building); and the rightmost number, an individual computer. The operating system uses the address in a different, lower-level form, converting it to its binary equivalent, a series of 1s and 0s. See the following Optional section for more information.

Static Versus Dynamic IP Addresses

A static IP address is one that remains the same. A dynamic IP address is one that can change each time you connect to the network. A dynamic address remains the same during a single login session. Any server (mail, Web, and so on) must have a static address so clients can find the machine that is the server. End-user machines usually work well with dynamic addresses. During a given login session, they can function as a client (your Web browser, for example) because they have a constant IP address. When you log out and log in again, it does not matter that you have a different IP address because your computer, acting as a client, establishes a new connection with a server. The advantage of dynamic addressing is that it allows inactive addresses to be reused, reducing the total number of IP addresses needed.

If a Mac OS X system is configured for a dynamic address but no server assigns it one, it assigns itself an address in the 169.254 block after a short delay. Refer to private address space (page 948) for information about addresses you can use on a LAN without registering them.

Optional: IP Classes

To facilitate routing on the Internet, IP addresses are divided into classes. These classes, which are labeled class A through class E, allow the Internet address space to be broken into blocks of small, medium, and large networks that are designed to be assigned based on the number of hosts within a network.

When you need to send a message to an address outside the local network, your system looks up the address block/class in its routing table and sends the message to the next router on the way to the final destination. Every router along the way does a similar lookup to forward the message. At the destination, local routers direct the message to the specific address. Without classes and blocks, your host would have to know every network and subnetwork address on the Internet before it could send a message. This setup would be impractical because of the large number of addresses on the Internet.

Each of the four numbers in the IP address is in the range 0255 because each segment of the IP address is represented by 8 bits (an octet), with each bit being capable of taking on two values; the total number of values is therefore 28 = 256. When you start counting at 0, the range 1256 becomes 0255.[7] Each IP address is divided into a net address (netid) portion, which is part of the class, and a host address (hostid) portion. See Table 10-2.

Table 10-2. IP classes

Class

Start bits

Address range

All bits (including start bits)

0-7

8-15

16-23

24-31

Class A

0

001.000.000.000-126.000.000.000

0-netid-

=========hostid=========

Class B

10

129.000.000.000-191.255.000.000

10-------netid-----

=====hostid=====

Class C

110

192.000.000.000-223.255.255.000

110----------netid------------

=hostid==

Class D (Multicast)

1110

224.000.000.000-239.255.255.000

1110

Class E (Reserved)

11110

240.000.000.000-255.255.255.000

11110

The first set of addresses, defining class A networks, is reserved for extremely large corporations, such as General Electric (3.0.0.0) and Hewlett-Packard (15.0.0.0), or for ISPs. One start bit (0) in the first position designates a class A network, 7 bits holds the network portion of the address (netid), and 24 bits holds the host portion of the address (hostid, Table 10-2). This setup means that GE can have 224, or approximately 16 million, hosts on its network. Unused address space and subnets (page 956) lower this number quite a bit. The 127.0.0.0 subnet (page 399) is reserved, as are several others (see private address space on page 948).

Two start bits (10) in the first two positions designates a class B network, 14 bits holds the network portion of the address (netid), and 16 bits holds the host portion of the address, for a potential total of 65,534 hosts.[8] A class C network uses 3 start bits (100), 21 netid bits (2 million networks), and 8 hostid bits (254 hosts). Today a new large customer will not receive a class A or B network but is likely to receive a class C or several (usually contiguous) class C networks, if merited.

Several other classes of networks exist. Class D networks are reserved for multicast (page 943) networks. When you run netstat nr on a Mac OS X system, you can see whether the machine is a member of a multicast network. A 224.0.0.0 in the Destination column that netstat displays indicates a class D, multicast address (Table 10-2). A multicast is like a broadcast, but only hosts that subscribe to the multicast group receive the message. To use Web terminology, a broadcast is like a push. A host pushes a broadcast on the network, and every host on the network must check each packet to see whether it contains relevant data. A multicast is like a pull. A host will see a multicast only if it registers itself as subscribed to a multicast group or service and pulls the appropriate packets from the network.

Table 10-3 shows some of the computations for IP address 131.204.027.027. Each address is shown in decimal, hexadecimal, and binary. Binary is the easiest to work with for bitwise (binary) computations. The first three lines show the IP address. The next three lines show the subnet mask (page 956) in three bases. Next the IP address and the subnet mask are ANDed together bitwise to yield the subnet number (page 956), which is shown in three bases. The last three lines show the broadcast address (page 924), which is computed by taking the subnet number and turning the hostid bits to 1s. The subnet number is the name/number of the local network. The subnet number and the subnet mask determine what range the IP address of the machine must be in. They are also used by routers to segment traffic; see network segment (page 944). A broadcast on this network goes to all hosts in the range 131.204.27.1 through 131.204.27.254 but will be acted on only by hosts that have a use for it.

Table 10-3. Computations for IP address 131.204.027.027

---------------Class B------------

netid

hostid

IP address

131

.204

.027

.027

decimal

83

CC

1B

1B

hexadecimal

1000 0011

1100 1100

0001 1011

0001 1011

binary

Subnet mask

255

.255

.255

.000

decimal

FF

FF

FF

00

hexadecimal

1111 1111

1111 1111

1111 1111

0000 0000

binary

IP address bitwise and subnet mask = subnet number

1000 0011

1100 1100

0001 1011

0001 1011

binary

1111 1111

1111 1111

1111 1111

0000 0000

1000 0011

1100 1100

0001 1011

0000 0000

Subnet number

131

.204

.027

.000

decimal

83

CC

1B

00

hexadecimal

1000 0011

1100 1100

0001 1011

0000 0000

binary

Broadcast address (Set host bits to 1)

131

.204

.27

.255

decimal

83

CC

1B

FF

hexadecimal

1000 0011

1100 1100

0001 1011

1111 1111

binary

SUBNETS

Each host on a network must process each broadcast packet to determine whether the information in the packet is useful to that host. If the network includes numerous hosts, each host must process many packets. To maintain efficiency, most networks, and particularly shared media networks such as Ethernet, need to be split into subnetworks, or subnets.[9] The more hosts on a network, the more dramatically network performance is affected. Organizations use router and switch technology called VLANs (virtual local area networks) to group similar hosts into broadcast domains (subnets) based on function. It is not uncommon to see a switch with different ports being part of different subnets. See page 453 for information on how to specify a subnet.

A subnet mask (or address mask) is a bit mask that identifies which parts of an IP address correspond to the network address and subnet portion of the address. This mask has 1s in positions corresponding to the network and subnet numbers and 0s in the host number positions. When you perform a bitwise AND on an IP address and a subnet mask (Table 10-3), the result is an address that contains everything except the host address (hostid) portion.

There are several ways to represent a subnet mask: A network could have a subnet mask of 255.255.255.0 (decimal), FFFFFF00 (hexadecimal), or /24 (the number of bits used for the subnet mask). If it were a class B network (of which 16 bits are already fixed), this yields 28 (24 total bits 16 fixed bits = 8 bits, 28 = 256) networks[10] with 28 2 (256 2 = 254) hosts[11] on each network.

For example, when you divide the class C address 192.25.4.0 into eight subnets, you get a subnet mask of 255.255.255.224, FFFFFFE0, or /27 (27 1s). The eight resultant networks are 192.25.4.0, 192.25.4.32, 192.25.4.64, 192.25.4.96, 192.25.4.128, 192.25.4.160, 192.25.4.192, and 192.25.4.224. You can use a Web-based subnet mask calculator to calculate subnet masks (refer to "Network Calculators" on page 913). To use this calculator to determine the preceding subnet mask, use an IP host address of 192.25.4.0.

CIDR: CLASSLESS INTER-DOMAIN ROUTING

CIDR (pronounced "cider") allows groups of addresses that are smaller than a class C block to be assigned to an organization or ISP and then further subdivided and parceled out. In addition, it helps to alleviate the potential problem of routing tables on major Internet backbone and peering devices becoming too large to manage.

The pool of available IPv4 addresses has been depleted to the point that no one gets a class A address anymore. The trend is to reclaim these huge address blocks, if possible, and recycle them into groups of smaller addresses. Also, as more class C addresses are assigned, routing tables on the Internet are filling up and causing memory overflows. The solution is to aggregate[12] groups of addresses into blocks and allocate them to ISPs, which in turn subdivide these blocks and allocate them to their customers. The address class designations (A, B, and C) described in the previous section are used less often today, although subnets are still encountered. When you request an address block, your ISP usually gives you as many addresses as you need and no more. The ISP aggregates several contiguous smaller blocks and routes them to your location. This aggregation is CIDR. Without CIDR, the Internet as we know it would not function.

For example, you might be allocated the 192.168.5.0/22 IP address block, which could support 210 hosts (32 22 = 10). Your ISP would set its routers so that any packets going to an address in that block would be sent to your network. Internally, your own routers might further subdivide this block of 1,024 potential hosts into subnets, perhaps into four networks. Four networks require an additional two bits of addressing (22 = 4). You could set up your router to have four networks with this allocation: 192.168.5.0/24, 192.168.6.0/24, 192.168.7.0/24, and 192.168.8.0/24. Each of these networks could have 254 hosts. CIDR lets you arbitrarily divide networks and subnetworks into ever smaller blocks along the way. Each router has enough memory to keep track of the addresses it needs to direct and aggregates the rest.

This scheme uses memory and address space efficiently. For example, you could take 192.168.8.0/24 and further divide it into 16 networks with 14 hosts each. The 16 networks require four more bits (24 = 16), so you would have 192.168.8.0/28, 192.168.8.16/28, 192.168.8.32/28, and so on, up through the last subnet of 192.168.8.240/16, which would have the hosts 192.168.8.241 through 192.168.8.254.

[7] Internally, the IP address is represented as a set of four unsigned 8-bit fields or a 32-bit unsigned number, depending on how programs are using it. The most common format in C is to represent it as a union of an unsigned 32-bit long integer, four unsigned chars, and two unsigned short integers.

[8] A 16-bit (class B) address can address 216 = 65,536 hosts, yet the potential number of hosts is two less than that because the first and last addresses on any network are reserved. In a similar manner, an 8-bit (class C) address can address only 254 hosts (28 2 = 254). The 0 host address (for example, 194.16.100.0 for a class C network or 131.204.0.0 for a class B network) is reserved as a designator for the network itself. Several older operating systems use this as a broadcast address. The 255 host address (for example, 194.16.100.255 for a class C network or 131.204.255.255 for a class B network) is reserved as the IP broadcast address. An IP packet (datagram) that is sent to this address is broadcast to all hosts on the network.

The netid portion of a subnet does not have the same limitations. Often you are given the choice of reserving the first and last networks in a range as you would a hostid, but this is rarely done in practice. More often the first and last networks in the netid range are used to provide more usable address space. Refer to "Subnets" on the following page.

[9] This is also an issue with other protocols, particularly AppleTalk.

[10] The first and last networks are reserved in a manner similar to the first and last host, although the standard is flexible. You can configure routers to reclaim the first and last networks in a subnet. Different routers have different techniques for reclaiming these networks.

[11] Subtract 2 because the first and last host addresses on every network are reserved.

[12] Aggregate means to join. In CIDR, the aggregate of 208.178.99.124 and 208.178.99.125 is 208.178.99.124/23 (the aggregation of two class C blocks).

Hostnames

People generally find it easier to work with names than with numbers, and Mac OS X provides several ways to associate hostnames with IP addresses. The oldest method is to consult a list of names and addresses that are stored in the /etc/hosts file:

$ cat /etc/hosts
127.0.0.1     localhost
130.128.52.1  gwexample.example.com  gwexample
130.128.52.2  bravo.example.com      bravo
130.128.52.3  hurrah.example.com     hurrah
130.128.52.4  kudos.example.com      kudos

localhost = 127.0.0.1

The address 127.0.0.1 is reserved for the special hostname localhost, which serves as a hook for the system's networking software to operate on the local machine without going onto a physical network. The names of the other systems are shown in two forms: in a fully qualified domain name (FQDN) format that is unique on the Internet and as a nickname that is locally unique.

NetInfo

As more hosts joined networks, storing these name-to-address mappings in a text file proved to be inefficient and inconvenient. The file grew ever larger and became impossible to keep up-to-date. Mac OS X supports NetInfo, which was developed for use on NeXT computers. NetInfo stores information in a database, making it easier to find a specific address, but it is useful only for host information within a single administrative domain. In many ways NetInfo is superior to flat text files. However, some hosts may not support NetInfo, and it does not scale to a global network with millions of hosts. See page 441 for more information on NetInfo.

The solution is DNS (Domain Name Service, page 411). DNS effectively addresses the efficiency and update issues by arranging the entire network namespace as a hierarchy. Each domain in the DNS manages its own namespace (addressing and name resolution), and each domain can easily query for any host or IP address by following the tree up or down the namespace until the appropriate domain is found. By providing a hierarchical naming structure, DNS distributes name administration across the entire Internet.

IPv6

The explosive growth of the Internet has uncovered deficiencies in the design of the current address planmost notably the shortage of addresses. Over the next few years, a revised protocol, named IPng (IP Next Generation), also known as IPv6 (IP version 6),[13] will be phased in (it may take longer; the phase-in is going quite slowly). This new scheme is designed to overcome the major limitations of the current approach and can be implemented gradually because it is compatible with the existing address usage. IPv6 makes it possible to assign many more unique Internet addresses (2128, or 340 undecillion [1036]) and supports security and performance control features.

[13] IPv5 referred to an experimental real-time stream protocol named STthus the jump from IPv4 to IPv6.

The IPv6 address plan

  • Enables autoconfiguration. With IPv4, autoconfiguration is available using optional DHCP (page 464). With IPv6, autoconfiguration is mandatory, making it easy for hosts to configure their IP addresses automatically.

  • Reserves 24 bits in the header for advanced services, such as resource reservation protocols, better backbone routing, and improved traffic engineering.

  • Makes multicast protocols mandatory and uses them extensively. In IPv4, multicast, which improves scalability, is optional.

  • Aggregates address blocks more efficiently because of the huge address space. This aggregation makes obsolete NAT (page 944), which decreased scalability and introduced protocol issues.

  • Provides a simplified packet header that allows hardware accelerators to work better.

A sample IPv6 address is fe80::a00:20ff:feff:5be2/10. Each group of four hexadecimal digits is equivalent to a number between 0 and 65,536 164). A pair of adjacent colons indicates a hex value of 0x0000; leading 0s need not be shown. With eight sets of hexadecimal groupings, there are 65,5368 = 2128 possible addresses. In an IPv6 address on a host with the default autoconfiguration, the first characters in the address are always fe80. The last 64 bits hold an interface ID designation, which is often the MAC address (page 941) of the Ethernet controller on the system.

Mac OS X supports IPv6, which can be configured through the Network preference pane in System Preferences. In rare cases you may need to disable IPv6 support on an interface due to compatibility issues.


Communicate over a Network

Many commands that you can use to communicate with other users on a single computer system have been extended to work over a network. Examples of extended utilities are electronic mail programs, information-gathering utilities (such as finger, page 61), and communications utilities. These utilities are examples of the UNIX philosophy: Instead of creating a new, special-purpose tool, modify an existing one.

Many utilities understand a convention for the format of network addresses: user@host (spoken as user at host). When you use an @ sign in an argument to one of these utilities, the utility interprets the text that follows as the name of a remote host. When it does not include an @ sign, a utility assumes that you are requesting information from or corresponding with someone on the LAN.

The prompts shown in the examples in this chapter include the hostname of the system you are using. When you frequently use more than one system over a network, you may find it difficult to keep track of which system you are using at any particular moment. The default prompt for bash under OS X includes the hostname of the local system so that it is always clear which system you are using. To identify the computer you are using, run hostname or uname n:

$ hostname
kudos.local

See page 286 for information on how you can change the prompt.

finger: Displays Information About Remote Users

The finger utility displays information about one or more users on a system. This utility was designed for local use, but when networks became popular, it was obvious that finger should be enhanced to reach out and collect information remotely. In the following examples, finger displays information about all users logged in on the system named bravo:

[kudos]$ finger @bravo
[bravo]
Login    Name                 TTY  Idle  Login  Time   Office  Phone
zach     Zach                *con  2:10  Sat    12:28
zach     Zach                 p1         Sat    13:17

A user's login name in front of the @ sign causes finger to display information from the remote system for the specified user only:

[kudos]$ finger zach@bravo
[bravo.example.com]
Login: zach                             Name: Zach
Directory: /Users/zach                  Shell: /bin/bash
On since Sat May 21 12:28 (CDT) on console, idle 2:12 (messages off)
On since Sat May 21 13:17 (CDT) on ttyp1 from grape.example.co
No Mail.
No Plan.

The finger utility works by querying a standard network service, the fingerd daemon, that runs on the system being queried. Although this service is supplied with Mac OS X, most sites choose not to run it to minimize the load on their systems, reduce security risks, or maintain privacy. It is disabled by default. When you use finger to obtain information about someone at such a site, you will see an error message or nothing at all. It is the remote fingerd daemon that determines how much information to share and in what format. As a result, the report displayed for any given system may differ from the preceding examples.

Security: The fingerd daemon

The finger daemon (fingerd) gives away system account information that can aid a malicious user. Some sites disable finger or randomize user account IDs to make a malicious user's job more difficult. This daemon is controlled through xinetd under Mac OS X version 10.3 and earlier and launchd under version 10.4 and later. It is disabled by default.

The information for remote finger looks much the same as it does when finger runs on the local system, with one difference: Before displaying the results, finger reports the name of the remote system that answered the query (bravo, as shown in brackets in the preceding example). The name of the host that answers may be different from the system name you specified on the command line, depending on how the finger daemon service is configured on the remote system. In some cases, several hostnames may be listed if one finger daemon contacts another to retrieve the information.

Sending Mail to a Remote User

Given a user's login name on a remote system and the name of the remote system or its domain, you can use an email program to send a message over the network or the Internet, using the @ form of an address:

jenny@bravo

or

jenny@example.com

Although the @ form of a network address is recognized by many Mac OS X utilities, you may find that you can reach more remote computers with email than with the other networking utilities described in this chapter. This disparity arises because the mail system can deliver a message to a host that does not run IP, even though it appears to have an Internet address. The message may be routed over the network, for example, until it reaches a remote system that has a point-to-point, dial-up connection to the destination system. Other utilities, such as talk, rely on IP and operate only between networked hosts.

Mailing List Servers

A mailing list server (listserv[14]) allows you to create and manage an email list. An electronic mailing list provides a means for people interested in a particular topic to participate in an electronic discussion and for a person to disseminate information periodically to a potentially large mailing list. One of the most powerful features of most list servers is their ability to archive email postings to the list, create an archive index, and allow users to retrieve postings from the archive based on keywords or discussion threads. Typically you can subscribe and unsubscribe from the list with or without human intervention. The owner of the list can restrict who can subscribe, unsubscribe, and post messages to the list. Popular list servers include LISTSERV (www.lsoft.com), Lyris (www.lyris.com), Majordomo (www.greatcircle.com/majordomo), Mailman (www.list.org), and ListProc (www.listproc.net). Apple maintains quite a few mailing lists and list archives for those lists (lists.apple.com). Use Google to search on OS X mailing list to find other lists.

[14] Although the term listserv is sometimes used generically to include many different list server programs, it is actually a specific product and a registered trademark of L-soft International, Inc.: LISTSERV (for information go to www.lsoft.com).


Network Utilities

To realize the full benefits of a networked environment, it made sense to extend certain tools, some of which have already been described. Networks also created a need for new utilities to control and monitor them, leading to ideas for new tools that took advantage of network speed and connectivity. This section describes concepts and utilities for systems attached to a network.

Trusted Hosts

Some commands, including rcp and rsh, work only if the remote system trusts your local computer (that is, the remote system knows your local computer and believes that it is not pretending to be another system). The /etc/hosts.equiv file lists trusted systems. For reasons of security, the Superuser account does not rely on this file to identify trusted Superusers from other systems.

Host-based trust is largely obsolete. Because there are many ways to subvert trusted host security, including subverting DNS systems and IP spoofing (page 938), authentication based on IP address is widely regarded as insecure and obsolete. In a small homogeneous network of machines with local DNS control, it can be "good enough." The ease of use in these situations may outweigh the security concerns. By default Mac OS X does not enable the services used by the insecure tools.

Security: Do not share your login account

You can use a .rhosts file to allow another user to log in as you from a remote system without knowing your password. This setup is not recommended. Do not compromise the security of your files or the entire system by sharing your login account. Use ssh and scp instead of rsh and rcp whenever possible.

OpenSSH Tools

The OpenSSH project provides a set of tools that replace rcp, rsh, and others with secure equivalents. These tools are installed by default and can be used as drop-in replacements for their insecure counterparts. Refer to scp (page 832) and ssh (page 847) for more information.

telnet: Logs In on a Remote System

You can use the TELNET protocol to interact with a remote computer. The telnet utility, a user interface to this protocol, is older than ssh and is not secure. Nevertheless, it may work where ssh is not available (there is more non-UNIX support for TELNET access than there is for ssh access). In addition, many legacy devices, such as terminal servers and network devices, do not support ssh.

[bravo]$ telnet kudos
Trying 172.19.52.2...
Connected to kudos.example.com
Escape character is '^]'.

Welcome to SuSE Linux 7.3 (i386) - Kernel 2.4.10-4GB (2).
kudos login: watson
Password:
You have old mail in /var/mail/watson.
Last login: Mon Feb 25 14:46:55 from bravo.example.com
watson@kudos:~>
...
watson@kudos:~> logout
Connection closed by foreign host.
[bravo]$

telnet versus ssh

When you connect to a remote UNIX system using telnet, you are presented with a regular, textual login: prompt. Unless you specify differently, the ssh utility assumes that your login name on the remote system matches that on the local system. Because telnet is designed to work with non-UNIX systems, it makes no such assumptions.

Security: telnet is not secure

Whenever you enter sensitive information, such as your password, while you are using telnet, it is transmitted in cleartext and can be read by someone who is listening in on the session.

Another difference between these two utilities is that telnet allows you to configure many special parameters, such as how RETURNs or interrupts are processed. When using telnet between two UNIX systems, you rarely need to change any parameters.

When you do not specify the name of a remote host on the command line, telnet runs in an interactive mode. The following example is equivalent to the previous telnet example:

[bravo]$ telnet
telnet> open kudos
Trying 172.19.52.2...
Connected to kudos.example.com
Escape character
is '^]'.
...

Before connecting you to a remote system, telnet tells you what the escape character is; in most cases, it is ^] (the ^ represents the CONTROL key). When you press CONTROL-], you escape to telnet's interactive mode. Continuing the preceding example:

[kudos]$ CONTROL-]
telnet> ?

(displays help information)

telnet> close
Connection closed.
[bravo]$

When you enter a question mark in response to the telnet> prompt, telnet lists its commands. The close command ends the current telnet session, returning you to the local system. To get out of telnet's interactive mode and resume communication with the remote system, press RETURN in response to a prompt.

You can use telnet to access special remote services at sites that have chosen to make such services available. However, many of these services, such as the U.S. Library of Congress Information System (LOCIS), have moved to the Web. As a consequence, you can now obtain the same information using a Web browser.

Using telnet to Connect to Other Ports

By default telnet connects to port 23, which is used for remote logins. However, you can use telnet to connect to other services by specifying a port number. In addition to standard services, many of the special remote services available on the Internet use unallocated port numbers. For example, you can access some multiplayer text games, called MUDs (Multi-User Dungeons, or Dimensions), using telnet to connect to a specified port, such as 4000 or 8888. Unlike the port numbers for standard protocols, these port numbers can be picked arbitrarily by the administrator of the game.

While telnet is no longer commonly employed to log in on remote systems, it is still used extensively as a debugging tool. This utility allows you to communicate directly with a TCP server. Some standard protocols are simple enough that an experienced user can debug problems by connecting to a remote service directly using telnet. For example, system administrators who are debugging problems with email delivery might use telnet to connect to the SMTP port (port 25) on a remote system. See page 870 for an example.

ftp: transfers Files over a Network

The File Transfer Protocol (FTP) is a method of downloading files from and uploading files to another system using TCP/IP over a network. FTP is not a secure protocol; use it only for downloading public information from a public server. Most Web browsers, including Safari, can download files from FTP servers directly.

ping: Tests a Network Connection

The ping[15] utility (http://ftp.arl.mil/~mike/ping.html) sends an ECHO_REQUEST packet to a remote computer. This packet causes the remote system to send back a reply. This exchange is a quick way to verify that a remote system is available as well as to check how well the network is operating, such as how fast it is or whether it is dropping data packets. The ping utility uses the ICMP (Internet Control Message Protocol) protocol. Without any options, ping tests the connection once per second until you abort the execution with CONTROL-C.

[15] The name ping mimics the sound of a sonar burst used by submarines to identify and communicate with each other. The word ping also expands to Packet Internet Groper.

$ ping tsx-11.mit.edu
PING tsx-11.mit.edu (18.7.14.121): 56 data bytes
64 bytes from 18.7.14.121: icmp_seq=0 ttl=49 time=82.108 ms
64 bytes from 18.7.14.121: icmp_seq=1 ttl=49 time=66.111 ms
64 bytes from 18.7.14.121: icmp_seq=2 ttl=49 time=65.200 ms
CONTROL-C

--- tsx-11.mit.edu ping statistics ---
4 packets transmitted, 3 packets received, 25% packet loss
round-trip min/avg/max/stddev = 65.200/71.140/82.108/7.765 ms

This example shows that the remote system named tsx-11.mit.edu is up and available over the network. The lost packet resulted from interrupting the command.

By default ping sends packets containing 64 bytes (56 data bytes and 8 bytes of protocol header information). In the preceding example, four packets were sent to the system tsx-11.mit.edu before the user interrupted ping by pressing CONTROL-C. The four-part number on each line is the remote system's IP address. A packet sequence number (called icmp_seq) is also given. If a packet is dropped, a gap occurs in the sequence numbers. The round-trip time is listed last, in milliseconds; it represents the time that elapsed from when the packet was sent from the local system to the remote system until the reply from the remote system was received by the local system. This time is affected by the distance between the two systems, network traffic, and the load on both computers. Before it terminates, ping summarizes the results, indicating how many packets were sent and received as well as the minimum, average, maximum, and standard deviation of the round-trip times it measured. Use the ping6 utility to test IPv6 networks.

Tip: When ping cannot connect

If it is unable to contact the remote system, ping continues trying until you interrupt it with CONTROL-C. A system may not answer for any of several reasons: The remote computer may be down, the network interface or some part of the network between the systems may be broken, a software failure may have occurred, or the remote machine may be set up, for reasons of security, not to return pings (TRy pinging www.microsoft.com or www.ibm.com).

traceroute: TRaces a Route over the Internet

The TRaceroute utility, which is supplied with Mac OS X, traces the route that an IP packet follows, including all intermediary points traversed (called network hops), to its destination (the argument to traceroutean Internet host). It displays a numbered list of hostnames, if available, and IP addresses, together with the round-trip time it took for a packet to get to each router along the way and an acknowledgment to get back. You can put this information to good use when you are trying to identify the location of a network bottleneck.

The traceroute utility has no concept of the path from one host to the next; it simply sends out packets with increasing TTL (time to live) values. TTL is an IP header field that indicates how many more hops the packet should be allowed to make before being discarded or returned. In the case of a traceroute packet, the packet is returned by the host that has the packet when the TTL value is zero. The result is a list of hosts that the packet travels through to get to its destination.

The TRaceroute utility can help you solve routing configuration problems and locate routing path failures. When you cannot reach a host, use TRaceroute to see what path the packet follows, how far it gets, and what the delay is.

The next example is the output of traceroute following a route from a local computer to www.linux.org. The first line tells you the IP address of the target, the maximum number of hops that will be traced, and the size of the packets that will be used. Each numbered line contains the name and IP address of the intermediate destination, followed by the time it takes a packet to make a round-trip to that destination and back. The traceroute utility sends three packets to each destination; thus there are three times on each line. Line 1 shows the statistics when a packet is sent to the local gateway (less than 3 milliseconds). Lines 46 show it bouncing around Mountain View (California) before it goes to San Jose. Between hops 13 and 14 the packet travels across the United States (San Francisco to somewhere in the East). By hop 18 the packet has found www.linux.org. The traceroute utility displays asterisks when it does not receive a response. Each asterisk indicates that TRaceroute has waited three seconds. Use traceroute6 to test IPv6 networks.

$ traceroute www.linux.org
traceroute to www.linux.org (198.182.196.56), 64 hops max, 40 byte packets
 1  gw.localco.com. (204.94.139.65)  2.904 ms  2.425 ms  2.783 ms
 2  covad-gw2.meer.net (209.157.140.1)  19.727 ms  23.287 ms  24.783 ms
 3  gw-mv1.meer.net (140.174.164.1)  18.795 ms  24.973 ms  19.207 ms
 4  d1-4-2.a02.mtvwca01.us.ra.verio.net (206.184.210.241)  59.091 ms d1-10-0-0-200.a03.
       mtvwca01.us.ra.verio.net (206.86.28.5)  54.948 ms  39.485 ms
 5  fa-11-0-0.a01.mtvwca01.us.ra.verio.net (206.184.188.1)  40.182 ms  44.405 ms 49.362 ms
 6  p1-1-0-0.a09.mtvwca01.us.ra.verio.net (205.149.170.66)  78.688 ms  66.266 ms 28.003 ms
 7  p1-12-0-0.a01.snjsca01.us.ra.verio.net (209.157.181.166) 32.424 ms 94.337 ms 54.946 ms
 8  f4-1-0.sjc0.verio.net (129.250.31.81)  38.952 ms  63.111 ms  49.083 ms
 9  sjc0.nuq0.verio.net (129.250.3.98)  45.031 ms  43.496 ms  44.925 ms
10  mae-west1.US.CRL.NET (198.32.136.10)  48.525 ms  66.296 ms  38.996 ms
11  t3-ames.3.sfo.us.crl.net (165.113.0.249)  138.808 ms  78.579 ms  68.699 ms
12  E0-CRL-SFO-02-E0X0.US.CRL.NET (165.113.55.2)  43.023 ms  51.910 ms  42.967 ms
13  sfo2-vva1.ATM.us.crl.net (165.113.0.254)  135.551 ms  154.606 ms  178.632 ms
14  mae-east-02.ix.ai.net (192.41.177.202)  158.351 ms  201.811 ms  204.560 ms
15  oc12-3-0-0.mae-east.ix.ai.net (205.134.161.2)  202.851 ms  155.667 ms  219.116 ms
16  border-ai.invlogic.com (205.134.175.254)  214.622 ms *  190.423 ms
17  router.invlogic.com (198.182.196.1)  224.378 ms  235.427 ms  228.856 ms
18  www.linux.org (198.182.196.56)  207.964 ms  178.683 ms 179.483 ms

host and dig: Query Internet Nameservers

The host utility looks up an IP address given a name, or vice versa. The following example shows how to use host to look up the domain name of a machine, given an IP address:

$ host 140.174.164.2
2.164.174.140.in-addr.arpa domain name pointer ns.meer.net.

You can also use host to determine the IP address of a domain name:

$ host ns.meer.net
ns.meer.net has address 140.174.164.2

The dig (domain information groper) utility queries DNS servers and individual machines for information about a domain. A powerful utility, dig has many features that you may never use. It is more complex than host.

whois: Looks Up Information About an Internet Site

The whois utility queries a whois server for information about an Internet site. This utility returns site contact and InterNIC or other registry information that can help you track down the person who is responsible for a site: Perhaps that person is sending you or your company spam (page 954). Many sites on the Internet are easier to use and faster than whois. Use a browser and search engine to search on whois or go to www.networksolutions.com/en_US/whois or www.ripe.net/perl/whois to get started.

When you search by name, whois may return more than one entry. In the following example, whois returns SOBELL.NET and SOBELL.COM when queried for sobell:

$ whois sobell

Whois Server Version 1.3

Domain names in the .com, .net, and .org domains can now be registered
with many different competing registrars. Go to http://www.internic.net
for detailed information.

SOBELL.NET
SOBELL.COM

To single out one record, look it up with "xxx", where xxx is one of the
of the records displayed above. If the records are the same, look them up
with "=xxx" to receive a full display for each record.

>>> Last update of whois database: Fri, 21 Jan 2005 07:45:07 EST <<<

...

The Registry database contains ONLY .COM, .NET, .ORG, .EDU domains and
Registrars.

When you do not specify a whois server and a hostname or IP address, as in the preceding example, whois defaults to whois.crsnic.net. When you specify a hostname or IP address, whois uses a more complicated algorithm, described in its man page, to try to find a whois server that can answer your query. Use the -h option to whois to specify a whois server.

To obtain information on a domain name, specify the complete domain name as in the following example:

$ Whois sobell.com
Whois Server Version 1.3

Domain names in the .com and .net domains can now be registered
with many different competing registrars. Go to http://www.internic.net
for detailed information

   Domain Name: SOBELL.COM
   Registrar: GO DADDY SOFTWARE, INC.
   Whois Server: whois.godaddy.com
   Referral URL: http://registrar.godaddy.com
   Name Server: NS.MEER.NET
   Name Server: NS2.MEER.NET
   Status: REGISTRAR-LOCK
   Updated Date: 10-nov-2004
   Creation Date: 07-apr-1995
   Expiration Date: 08-apr-2013

 >>> Last update of whois database: Tue, 21 Dec 2004 19:15:10 EST <<<

 ...

 Please note: the registrant of the domain name is specified
 in the "registrant" field. In most cases, Go Daddy Software, Inc.
 is not the registrant of domain names listed in this database.

 Registrant:
    Sobell Associates Inc
    POBox 460068
    San Francisco, California 94146-0068
    United States

    Registered through: GoDaddy.com
    Domain Name: SOBELL.COM
       Created on: 07-Apr-95
       Expires on: 08-Apr-13
       Last Updated on: 16-Jan-04

    Administrative Contact:
       Sobell, Mark  sobell@meer.net
       Sobell Associates Inc
       PO BOX 460068
       SAN FRANCISCO, California 94146-0068
       United States
       9999999999     Fax -- 9999999999
    Technical Contact:
       ,  hostmaster@meer.net
       meer.net
       po box 390804
       Mountain View, California 94039
       United States
       18888446337      Fax -- 18888446337

    Domain servers in listed order:
       NS.MEER.NET
       NS2.MEER.NET

Several top-level registries serve various regions of the world. You are most likely to use the following ones:

North American Registry | whois.arin.net
European Registry | www.ripe.net
Asia-Pacific Registry | www.apnic.net
American Military | whois.nic.mil
American Government | www.nic.gov


Distributed Computing

When many similar systems are on a network, it is often desirable to share common files and utilities among them. For example, a system administrator might choose to keep a copy of the system documentation on one computer's disk and to make those files available to remote systems. In this case, the system administrator configures the files so users who need to access the online documentation are not aware that the files are stored on a remote system. This type of setup, which is an example of distributed computing, not only conserves disk space but also allows you to update one central copy of the documentation rather than tracking down and updating copies scattered throughout the network on many different systems.

Figure 10-2 illustrates a fileserver that stores the system manual pages and users' home directories. With this arrangement, a user's files are always available to that userno matter which system the user logs in on. Each system's disk might contain a directory to hold temporary files as well as a copy of the operating system. Apple encourages the use of Apple Filing Protocol (AFP) instead of NFS for file sharing.

Figure 10-2. A fileserver

The Client/Server Model

Mainframe model

The client/server model was not the first computational model. First came the mainframe, which follows a one-machine-does-it-all model. That is, all the intelligence is in one system, including the data and the program that manipulates and reports on the data. Users connect to a mainframe using terminals.

File-sharing model

With the introduction of PCs, file-sharing networks became available. Data was downloaded from a shared location to a user's PC, where a program then manipulated the data. The file-sharing model ran into problems as networks expanded and more users needed access to the data.

Client/server model

In the client/server model, a client uses a protocol, such as FTP, to request services, and a server provides the services that the client requests. Rather than providing data files as the file-sharing model does, the server in a client/server relationship is a database that provides only those pieces of information that the client needs or requests.

The client/server model dominates UNIX and Mac OS X system networking and underlies most of the network services described in this book. FTP, NFS, DNS, email, and HTTP (the Web browsing protocol) all rely on the client/server model. Some servers, such as Web servers and browser clients, are designed to interact with specific utilities. Other servers, such as those supporting DNS, communicate with one another, in addition to answering queries from a variety of clients. Clients and servers can reside on the same or different systems running the same or different operating systems. The systems can be proximate or thousands of miles apart. A system that is a server to one system can turn around and act as a client to another. A server can reside on a single system or, as is the case with DNS, be distributed among thousands of geographically separated systems running many different operating systems.

Peer-to-peer model

Contrast the client/server model to the peer-to-peer (PTP) model, in which either program can initiate a transaction. PTP protocols are common on small networks. Microsoft's Network Neighborhood and Apple's AppleTalk both rely on broadcast-based PTP protocols for browsing and automatic configuration. The Zeroconf multicast DNS protocol is a PTP alternative DNS for small networks. The highest-profile PTP networks are those used for file sharing, such as Kazaa and GNUtella. Many of these networks are not pure PTP topologies. Pure PTP networks do not scale well, so networks such as Napster and Kazaa employ a hybrid approach.

DNS: Domain Name Service

DNS is a distributed service: Nameservers on thousands of machines around the world cooperate to keep the database up-to-date. The database itself, which contains the information that maps hundreds of thousands of alphanumeric hostnames to numeric IP addresses, does not exist in one place. That is, no system has a complete copy of the database. Instead, each system that runs DNS knows about the hosts that are local to that site and understands how to contact other nameservers to learn about other, nonlocal hosts.

Like the Mac OS X filesystem, DNS is organized hierarchically. Each country has an ISO (International Organization for Standardization) country code designation as its domain name. (For example, AU represents Australia, IL is Israel, and JP is Japan; see www.iana.org/cctld/cctld.htm for a complete list.) Although the United States is represented in the same way (US) and uses the standard two-letter Postal Service abbreviations to identify the next level of the domain, only governments and a few organizations use these codes. Schools in the US domain are represented by a third- (and sometimes second-) level domain: k12. For example, the domain name for Myschool in New York state could be www.myschool.k12.ny.us.

Following is a list of the six original top-level domains. These domains are used extensively within the United States and, to a lesser degree, by users in other countries:

COM | Commerical enterprises
EDU | Educational institutions
GOV | Nonmilitary government agencies
MIL | Military government agencies
NET | Networking organizations
ORG | Other (often nonprofit) organizations

As this book was being written, the following additional top-level domains had been approved for use:

AERO | Air-transport industry
BIZ | Business
COOP | Cooperatives
INFO | Unrestricted use
MUSEUM | Museums
NAME | Name registries

Like Internet addresses, domain names were once assigned by the Network Information Center (NIC, page 393). Now they are assigned by several companies. A system's full name, referred to as its fully qualified domain name (FQDN), is unambiguous in the way that a simple hostname cannot be. The system okeeffe.berkeley.edu at the University of California at Berkeley (Figure 10-3) is not the same as one named okeeffe.moma.org, which might represent a host at the Museum of Modern Art. The domain name not only tells you something about where the system is located but also adds enough diversity to the namespace to avoid confusion when different sites choose similar names for their systems.

Figure 10-3. Top-level domains

Unlike the filesystem hierarchy, the top-level domain name appears last (reading from left to right). Also, domain names are not case sensitive. The names okeeffe.berkeley.edu, okeeffe.Berkeley.edu, and okeeffe.Berkeley.EDU refer to the same computer. Once a domain has been assigned, the local site is free to extend the hierarchy to meet local needs.

With DNS, mail addressed to user@example.com can be delivered to the computer named example.com that handles the corporate mail and knows how to forward messages to user mailboxes on individual machines. As the company grows, the site administrator might decide to create organizational or geographical subdomains. The name delta.ca.example.com might refer to a system that supports California offices, with alpha.co.example.com dedicated to Colorado. Functional subdomains might be another choice, with delta.sales.example.com and alpha.dev.example.com representing the sales and development divisions, respectively.

BIND

On Mac OS X systems, the lookupd daemon handles lookup requests, often delegating them to other servers. A system that is acting as a nameserver will run BIND (Berkeley Internet Name Domain). BIND follows the client/server model. On any given local network, one or more systems may be running a nameserver, supporting all the local hosts as clients. When it wants to send a message to another host, a system queries the nearest nameserver to learn the remote host's IP address. The client, called a resolver, may be a process running on the same computer as the nameserver, or it may pass the request over the network to reach a server. To reduce network traffic and accelerate name lookups, the local nameserver has some knowledge of distant hosts. If the local server must contact a remote server to pick up an address, when the answer comes back, the local server adds that address to its internal table and reuses it for a while. The nameserver deletes the nonlocal information before it can become outdated. Refer to "TTL" on page 958.

How the system translates symbolic hostnames into addresses is transparent to most users; only the system administrator of a networked system needs to be concerned with the details of name resolution. Systems that use DNS for name resolution are generally capable of communicating with the greatest number of hostsmore than would be practical to maintain in a /etc/hosts file or private NetInfo database (page 441).

Three common sources are referenced for hostname resolution: NetInfo, DNS, and system files (such as /etc/hosts). The default behavior is to check the system files first, then Netinfo, then DNS. This behavior is controlled by the lookupd daemon. The lookupd man page explains how you can change the behavior.

Ports

Ports are logical channels on a network interface and are numbered from 1 to 65,535. Each network connection is uniquely identified by the IP address and port number of each endpoint.

In a system that has many network connections open simultaneously, the use of ports keeps packets (page 946) flowing to and from the appropriate programs. A program that needs to receive data binds to a port and then uses that port for communication.

Privileged ports

Services are associated with specific ports, generally with numbers less than 1,024. These ports are called privileged (or reserved) ports. For security reasons, only root can bind to privileged ports. A service run on a privileged port provides assurance that the service is being provided by someone with authority over the system, except that any user on Windows 98 and earlier Windows systems can bind to any port. Commonly used ports include 22 (SSH), 23 (TELNET), 80 (HTTP), 111 (Sun RPC), and 201208 (AppleTalk).

NFS: Network Filesystem

The NFS (Network Filesystem) protocol allows a server to share selected local directory hierarchies with client systems on a heterogeneous network. Files on the remote fileserver appear as if they are present on the local system.

Optional: Internet Services

Mac OS X Internet services are provided by daemons that run continuously or by a daemon that is started automatically by launchd (page 460). Under OS X version 10.3 and earlier, daemons are started by xinetd (page 459). The xinetd and launchd daemons are called superservers (page 456). The /etc/services file lists network services (for example, telnet, ftp, ssh) and their associated numbers. Any service that uses TCP/IP or UDP/IP has an entry in this file. IANA (Internet Assigned Numbers Authority) maintains a database of all permanent, registered services. The /etc/services file usually lists a small, commonly used subset of services. Visit www.rfc.net/rfc1700.html for more information and a complete list of registered services.

Most of the daemons (the executable files) are stored in /usr/sbin or /usr/libexec. By convention, the names of many daemons end with the letter d to distinguish them from utilities (one common daemon whose name does not end in d is sendmail). The prefix rpc. is often used for daemon names. Look at /usr/sbin/*d to see a list of many of the daemon programs on the local system. Refer to "Startup Scripts" on page 436 for information about starting and stopping these daemons.

As an example of how a daemon works, consider what happens when you run ssh. The local system contacts the ssh daemon (sshd) on the remote system to establish a connection. The two systems negotiate the connection according to a fixed protocol. Each system identifies itself to the other, and then they take turns asking each other specific questions and waiting for valid replies. Each network service follows its own protocol.

In addition to the daemons that support the utilities described up to this point, many other daemons support system-level network services that you will not typically interact with. Some of these daemons are listed in Table 10-4. Not all of these daemons are enabled by default under Mac OS X. Some must be enabled from the command line; they cannot be enabled using the graphical system administration tools.

Table 10-4. Common daemons

Daemon

Used for or by

Function

cron

Run at system startup

Used for periodic execution of tasks, this daemon looks in the /var/cron/tabs directory for files with filenames that correspond to users' login names. It also looks at the /etc/crontab file. When a task comes up for execution, cron executes it as the user who owns the file that describes the task.

fingerd

finger

Handles requests for user information from the finger utility. Launched by launchd.

ftpd

FTP

Handles FTP requests. Refer to "ftp: Transfers Files over a Network" on page 405. Launched by launchd.

httpd

HTTP

The Web server daemon (Apache).

launchd

Internet superserver

Listens for service requests on network connections and starts up the appropriate daemon to respond to any particular request. Because of launchd, a system does not need the daemons running all the time to handle various network requests. For more information refer to "The launchd Superserver" on page 456. In Mac OS X 10.3 and earlier, daemons are launchd by xinetd.

ntpd

NTP

Synchronizes time on network computers. It requires a /etc/ntp.conf file. For more information go to www.ntp.org. The Date & Time preferences in System Preferences can configure ntpd automatically through the Set date & time automatically check box.

portmap

RPC

Maps incoming requests for RPC service numbers to TCP or UDP port numbers on the local system. Refer to "RPC Network Services" on page 417.

postfix

Mail programs

Replaces the sendmail daemon and does the same things sendmail does on other systems. The postfix daemon always listens on port 25 for incoming mail connections and then calls a local delivery agent, such as /bin/mail.

rexecd

rexec

Allows a remote user with a valid username and password to run programs on a system. Its use is generally deprecated for security reasons, but certain programs, such as PC-based X servers, may still have it as an option. Launched by launchd, but normally disabled.

smbd, nmbd

Samba

Allow Windows PCs to share files and printers with Mac OS X computers.

sshd

ssh, scp

Enables secure logins between remote systems.

syslogd

System log

Transcribes important system events and stores them in files and/or forwards them to users or another host running the syslogd daemon. This daemon is configured with /etc/syslog.conf and used with the syslog or logger utilities.

talkd

talk

Allows you to have a conversation with another user on the same or a remote system. The talkd daemon handles the connections between the systems. The talk utility on each system contacts the talkd daemon on the other system to set up a bidirectional conversation. Launched by the superserver.

telnetd

TELNET

One of the original Internet remote access protocols (page 404). Launched by launchd, but normally disabled.

xinetd

Internet superserver

Listens for service requests on network connections and starts up the appropriate daemon to respond to any particular request. Because of xinetd, a system does not need the daemons running all the time to handle various network requests. For more information refer to "The xinetd Superserver" on page 459. Under Mac OS X 10.4 and later, xinetd is replaced by launchd.

PROXY SERVERS

A proxy is a network service that is authorized to act for a system while not being part of that system. A proxy server or proxy gateway provides proxy services; it is a transparent intermediary, relaying communications back and forth between an application, such as a browser and a server, usually outside of a LAN and frequently on the Internet. When more than one process uses the proxy gateway/server, it must keep track of which processes are connecting to which hosts/servers so that it can route the return messages to the proper process. The most common proxies you will encounter are email and Web proxies.

A proxy server/gateway insulates the local computer from all other computers or from specified domains by using at least two IP addresses: one to communicate with your local computer and one to communicate with a server. The proxy server/gateway examines and changes the header information on all packets it handles so that it can encode, route, and decode them properly. The difference between a proxy gateway and a proxy server is that the proxy server usually includes cache (page 925) to store frequently used Web pages so that the next request for that page is available locally and quickly; a proxy gateway usually does not use cache. The terms proxy server and proxy gateway are frequently interchanged.

Proxy servers/gateways are available for such common Internet services as HTTP, HTTPS, FTP, SMTP, and SNMP. When an HTTP proxy sends queries from local systems, it presents a single organizationwide IP address (the external IP address of the proxy server/gateway) to all servers. It funnels all user requests to servers and keeps track of them. When the responses come back, it fans them out to the appropriate applications by using each system's unique IP address, thereby protecting local addresses from remote/specified servers.

Proxy servers/gateways are generally just one part of an overall firewall strategy to prevent intruders from stealing information or damaging an internal network. Other functions that can be combined with or kept separate from the proxy server/gateway include packet filtering, which blocks traffic based on origin and type, and user activity reporting, which helps management learn how the Internet is being used.

RPC NETWORK SERVICES

Much of the client/server interaction over a network is implemented using the RPC (Remote Procedure Call) protocol, which is implemented as a set of library calls that make network access transparent to the client and server. RPC specifies and interprets messages and does not concern itself with transport protocols; it runs on top of TCP/IP and UDP/IP. Services that use RPC include NFS and NIS. RPC was developed by Sun as ONC RPC (Open Network Computing Remote Procedure Calls) and differs from Microsoft RPC.

In the client/server model, a client contacts a server on a specific port (page 414) to avoid any mixup between services, clients, and servers. To avoid maintaining a long list of port numbers and to enable new clients/servers to start up without registering a port number with a central registry, when a server that uses RPC starts, it specifies the port it expects to be contacted on. Common RPC servers have port numbers that are defined by Sun. If a server does not use a predefined port number, it picks an arbitrary number.

The server then registers this port with the RPC portmapper (the portmap daemon) on the local system. The server tells the daemon which port number it is listening on and which RPC program numbers it serves. As a result of these exchanges, the portmap daemon knows the location of every registered port on the host and the programs that are available on each port. The portmap daemon, which always listens on port 111 for both TCP and UDP, must be running to make RPC calls. The portmap daemon is started automatically when RPC-based services are enabled through the system's GUI.

Files

The /etc/rpc file (page 446) maps RPC services to RPC numbers. The /etc/services file (page 446) lists system services.

RPC client/server communication

The sequence of events for communication between an RPC client and server is as follows:

  1. The client program on the client system makes an RPC call to obtain data from a (remote) server system. (The client issues a read record from a file request.)

  2. If RPC has not yet established a connection with the server system for the client program, it contacts portmap on port 111 of the server and asks which port the desired RPC server is listening on (for example, rpc.nfsd).

  3. The portmap daemon on the remote server looks in its tables and returns a UDP or TCP port number to the local system, the client (typically 2049 for nfs).

  4. The RPC libraries on the server system receive the call from the client and pass the request to the appropriate server program. The origin of the request is transparent to the server program. (The filesystem receives the read record from file request.)

  5. The server responds to the request. (The filesystem reads the record.)

  6. The RPC libraries on the remote server return the result over the network to the client program. (The read record is returned to the calling program.)

Because standard RPC servers are normally started by xinetd (page 459) on systems running Mac OS 10.3 and earlier, the portmap daemon must be started before the xinetd daemon is invoked if any RPC servers are enabled. Under Mac OS X 10.4 and later, the portmap daemon is started automatically if the system can determine that it is required. If the portmap daemon stops, you must restart all RPC servers on the local system.


Usenet

One of the earliest information services available on the Internet, Usenet is an electronic bulletin board that allows users with common interests to exchange information. Usenet is an informal, loosely connected network of systems that exchange email and news items (commonly referred to as netnews). It was formed in 1979 when a few sites decided to share some software and information on topics of common interest. They agreed to contact one another and to pass the information along over dial-up telephone lines (at that time running at 1,200 baud at best), using UNIX's uucp utility (UNIX-to-UNIX copy program).

The popularity of Usenet led to major changes in uucp to handle the ever-escalating volume of messages and sites. Today much of the news flows over network links using a sophisticated protocol designed especially for this purpose: NNTP (Network News Transfer Protocol). The news messages are stored in a standard format, and the many public domain programs available let you read them. An old, simple interface is named readnews. Others, such as rn, its X Window System cousin xrn, tin, nn, and xvnews have many features that help you browse through and reply to the articles that are available or create articles of your own. In addition, Netscape and Mozilla include an interface that you can use to read news (Netscape/Mozilla News) as part of their Web browsers. One of the easiest ways to read netnews is to go to groups.google.com.

The program you select to read netnews is largely a matter of personal taste. Some Mac-specific graphical clients, however, do not use X11. Mac OS X does not come with command line Usenet client programs installed.

Because programs to read netnews articles have been ported to non-UNIX systems, the community of netnews users has become highly diversified. In the UNIX tradition, categories of netnews groups are structured hierarchically. The top level includes such designations as comp (computer-related), misc (miscellaneous), rec (recreation), sci (science), soc (social issues), and talk (ongoing discussions). Usually at least one regional category is at the top level, such as ba (San Francisco Bay Area), and includes information about local events. New categories are continually being added to the more than 30,000 newsgroups. The names of newsgroups resemble domain names but are read from left to right (like UNIX filenames): comp.os.unix.misc, comp.lang.c, misc.jobs.offered, rec.skiing, sci.med, soc.singles, and talk.politics are but a few examples. The following article appeared in comp.sys.mac.system:

> I'm on a 700Mhz G4 eMac with 10.4.1. Ever since I got
> 10.4.1 I can't properly Shut Down.  It won't go all the way
> off.  A message comes on in several languages saying I must
> restart the computer by holding down the power button for
> several seconds

You are having a kernel panic. Is there any interesting hardware
connected to this machine? Does this same problem occur if you hold the
Shift key down throughout the startup process (called "safe boot mode")
and then shut down? m.

A great deal of useful information is available on Usenet, but you need patience and perseverance to find what you are looking for. You can ask a question, as the user did in the previous example, and someone from halfway around the world may answer it. Before posing such a simple question and causing it to appear on thousands of systems around the world, however, ask yourself whether you can get help in a less invasive way. Try the following:

  • Refer to the man pages and info.

  • Look through the files in /usr/share/doc.

  • Ask the system administrator or another user for help.

  • All of the popular newsgroups have FAQs (lists of frequently asked questions). Consult these lists and see whether your question has been answered. FAQs are periodically posted to the newsgroups; in addition, all of the FAQs are archived at sites around the Internet, including the Usenet newsgroup comp.answers and ftp://ftp.uu.net (see the index/info directory for a list).

  • Because someone has probably asked the same question earlier, search the netnews archives for an answer. Try looking at groups.google.com, which has a complete netnews archive.

  • Use a search engine to find an answer. One good way to get help is to search on an error message.

  • Review support documents at www.apple.com/support.

  • Contact a Macintosh users' group.

Post a query to the worldwide Usenet community as a last resort. If you are stuck on a Mac OS X question and cannot find any other help, try submitting it to one of these newsgroups:

  • comp.sys.mac.system

  • comp.sys.mac.general

One way to find out about new tools and services is to read Usenet news. The comp.sys.mac hierarchy is of particular interest to Mac OS X users; for example, news about newly released software for Mac OS X is posted to comp.sys.mac.announce. People often announce the availability of free software there, along with instructions on how to get a copy for your own use using anonymous FTP. Other tools to help you find resources, both old and new, exist on the network; see Appendix B.


WWW: World Wide Web

The World Wide Web (WWW, W3, or the Web) provides a unified, interconnected interface to the vast amount of information stored on computers around the world. The idea that spawned the World Wide Web came from the mind of Tim Berners-Lee of the European Particle Physics Laboratory (CERN) in response to a need to improve communications throughout the High Energy Physics community. The first generation was a notebook program named Enquire, short for Enquire Within Upon Everything (the name of a book from Berners-Lee's childhood), which he created in 1980 on a NeXT computer and which provided for links to be made between named nodes. It was not until 1989 that the concept was proposed as a global hypertext project to be known as the World Wide Web. In 1990, Berners-Lee wrote a proposal for a hypertext project, which eventually produced HTML (Hypertext Markup Language), the common language of the Web. The World Wide Web program became available on the Internet in the summer of 1991. By designing the tools to work with existing protocols, such as FTP and gopher, the researchers who created the Web created a system that is generally useful for many types of information and across various types of hardware and operating systems.

The WWW is another example of the client/server paradigm. You use a WWW client application, or browser, to retrieve and display information stored on a server that may be located anywhere on your local network or the Internet. WWW clients can interact with many types of servers. For example, you can use a WWW client to contact a remote FTP server and display the list of files it offers for anonymous FTP. Most commonly you use a WWW client to contact a WWW server, which offers support for the special features of the World Wide Web that are described in the remainder of this chapter.

The power of the Web derives from its use of hypertext, a way to navigate through information by following cross-references (called links) from one piece of information to another. To use the Web effectively, you need to be able to run interactive network applications. The first GUI for browsing the Web was a tool named Mosaic, which was released in February 1993. It was designed at the National Center for Supercomputer Applications at the University of Illinois and sparked a dramatic increase in the number of users of the World Wide Web. Marc Andreessen, who participated in the Mosaic project at the University of Illinois, later cofounded Netscape Communications with the founder of Silicon Graphics, Jim Clark. They created Netscape Navigator, a Web client program that was designed to perform better and support more features than the Mosaic browser. Netscape Navigator has enjoyed immense success and has become a popular choice for users exploring the World Wide Web. Important for Mac OS X users is the fact that from the beginning Netscape has provided versions of its tools that run on the Mac. Also, Netscape created Mozilla (mozilla.org) as an open-source browser project. Apple's Safari browser is based on Konqueror's (www.konqueror.org) rendering engine, the code used to display Web pages.

These browsers provide GUIs that allow you to listen to sounds, watch Web events or live news reports, and display pictures as well as text, giving you access to hypermedia. A picture on your screen may be a link to more detailed, nonverbal information, such as a copy of the same picture at a higher resolution or a short animation. If your system can produce audio output, you can listen to audio clips that have been linked to a document.

URL: Uniform Resource Locator

Consider the URL http://www.w3.org/pub/WWW. The first component in the URL indicates the type of resource, in this case http (HTTPHypertext Transfer Protocol). Other valid resource names, such as https (HTTPSsecure HTTP) and ftp (FTPFile Transfer Protocol), represent information available on the Web using other protocols. Next comes a colon and double slash (://). Frequently the http:// string is omitted from a URL in print, as you seldom need to enter it to get to the URL. Following this is the full name of the host that acts as the server for the information (www.w3.org/). The rest of the URL is a relative pathname to the file that contains the information (pub/WWW). Enter a URL in the location bar text box of a Web browser, and the Web server returns the page, frequently an HTML (page 936) file, pointed to by this URL.

By convention, many sites identify their WWW servers by prefixing a host or domain name with www. For example, you can reach the Web server at the New Jersey Institute of Technology at www.njit.edu. When you use a browser to explore the World Wide Web, you may never need to use a URL directly. However, as more information is published in hypertext form, you cannot help but find URLs everywherenot just online in email messages and Usenet articles, but also in newspapers, in advertisements, and on product labels.

Browsers

Most Macintosh users are familiar with Apple's Safari Web browser. Mozilla, Firefox, Opera, and several other browsers are also available for free. In addition, you can purchase commercial browsers, such as OmniWeb (www.omnigroup.com), that will run under OS X.

Search Engines

Search engine is a name that applies to a group of hardware and software tools that help you find World Wide Web sites that have the specific information you are looking for. A search engine relies on a database of information collected by a Web crawler, a program that regularly looks through the millions of pages that make up the World Wide Web. A search engine must also have a way of collating the information the Web crawler collects so that you can access it quickly, easily, and in a manner that makes it most useful to you. This part of the search engine, called an index, allows you to search for a word, a group of words, or a concept and returns the URLs of Web pages that pertain to what you are searching for. Many different types of search engines are available on the Internet, each with its own set of strengths and weaknesses.


Chapter Summary

A Mac OS X system attached to a network is probably communicating on an Ethernet, which may in turn be linked to other local area networks (LANs) and wide area networks (WANs). Communication between LANs and WANs requires the use of gateways and routers. Gateways translate the local data into a format suitable for the WAN, and routers make decisions about the optimal routing of the data along the way. The most widely used network, by far, is the Internet.

Basic networking tools allow Mac OS X users to log in and run commands on remote systems (ssh, telnet) and copy files quickly from one system to another (scp, ftp/sftp). Many tools that were originally designed to support communication on a single-host computer (for example, finger and talk) have been extended to recognize network addresses, thus allowing users on different systems to interact with one another. Other features, such as the Network Filesystem (NFS), were created to extend the basic UNIX model and to simplify information sharing.

Concern is growing about our ability to protect the security and privacy of machines connected to networks and of data transmitted over networks. Toward this end, many new tools and protocols have been created: ssh, scp, HTTPS, IPv6, firewall hardware and software, VPN, and so on. Many of these tools take advantage of newer, more impenetrable encryption techniques. In addition, some weaker concepts, such as that of trusted hosts, and some tools, such as finger and rwho, are being discarded in the name of security.

Two major advantages of computer networks over other ways of connecting computers are that they enable systems to communicate at high speeds and that they require few physical interconnections (typically one per system, often on a shared cable). The Internet Protocol (IP), the universal language of the Internet, has made it possible for dissimilar computer systems around the world to communicate easily with one another. Technological advances continue to improve the performance of computer systems and the networks that link them.

One way to gather information on the Internet is via Usenet. Many Mac OS X users routinely read Usenet news (netnews) to learn about the latest resources available for their systems. Usenet news is organized into newsgroups that cover a wide range of topics, computer related and otherwise. To read Usenet news, you need to have access to a news server and the appropriate client software. Many modern mailers, such as Mozilla and Netscape, can display netnews.

The rapid increase of network communication speeds in recent years has encouraged the development of many new applications and services. The World Wide Web provides access to vast information stores on the Internet and makes extensive use of hypertext links to promote efficient searching through related documents. It adheres to the client/server model so pervasive in networking. Typically the WWW client is local to a site or is made available through an Internet service provider. WWW servers are responsible for providing the information requested by their many clients.

Safari is a WWW client program that has enormous popular appeal. Safari and other browsers use a GUI to give you access to text, picture, and audio information: Making extensive use of these hypermedia simplifies access to and enhances the presentation of information.


Exercises

1.

Describe the similarities and differences among these utilities:

  1. scp and ftp

  2. ssh and telnet

  3. rsh and ssh

2.

Assuming rwho is disabled on the systems on your LAN, describe two ways to find out who is logged in on some of the other machines attached to your network.

3.

Explain the client/server model and give three examples of services on Mac OS X systems that take advantage of this model.

4.

A software implementation of chess was developed by GNU and is free software. How can you use the Internet to find a copy and download it?

5.

What is the difference between the World Wide Web and the Internet?

6.

If you have access to the World Wide Web, answer the following:

  1. Which browser do you use?

  2. What is the URL of the author of this book's home page? How many links does it have?

  3. Does your browser allow you to create bookmarks? If so, how do you create a bookmark? How can you delete one?

7.

Give one advantage and two disadvantages of using a wireless network.


Advanced Exercises

8.

Suppose the link between routers 1 and 2 is down in the Internet shown in Figure 10-1 on page 390. What happens if someone at site C sends a message to a user on a workstation attached to the Ethernet cable at site A? What happens if the router at site A is down? What does this tell you about designing network configurations?

9.

If you have a class B network and want to divide it into subnets, each with 126 hosts, which subnet mask should you use? How many networks will be available? What are the four addresses (broadcast and network number) for the network starting at 131.204.18?

10.

Suppose you have 300 hosts and want to have no more than about 50 hosts per subnet. What size address block should you request from your ISP? How many class Cequivalent addresses would you need? How many subnets would you have left over from your allocation?

11.

On your system, find two daemons running that are not listed in this chapter and explain what purpose they serve.

Review what services/daemons are automatically started on your system, and consider which you might turn off. Are there any services/daemons in the list in Table 10-4 on page 415 that you would consider adding?


Chapter 11. System Maintenance

Previous Page
Next Page

11. System Maintenance

IN THIS CHAPTER

System Administrator and Superuser | 427
System Maintenance Tools | 430
sudo: Performs Specific Tasks as Another User | 431
System Operation | 435
Booting the System | 436
Important Standard Directories and Files | 444
lookupd: Which Service to Look at First | 455
The Superserver | 456
DHCP | 464
PAM | 466
fink: Downloads and Installs Software | 470

The job of a system administrator, or maintainer, is to keep one or more systems useful and convenient for users. On a Mac OS X system, the administrator and user may both be you, with you and a single computer only a few feet apart. Alternatively, the system administrator may be halfway around the world, supporting a network of systems, with you simply one of thousands of users. A system administrator can be one person who works part-time taking care of a system and perhaps is also a user of the system. Alternatively, the administrator can be several people, all working full-time to keep many systems running.

A well-maintained system

  • Runs quickly enough that users do not get too frustrated waiting for the system to respond or complete a task.

  • Has enough storage to accommodate users' reasonable needs.

  • Provides a working environment appropriate to each user's abilities and requirements.

  • Is protected from malicious and accidental acts that might potentially alter its performance or compromise the security of the data it holds and exchanges with other systems.

  • Is backed up regularly with recently backed-up files readily available to users.

  • Has recent copies of the software that users need to get their jobs done.

  • Is easier to administer than a poorly maintained system.

In addition, a system administrator should be available to help users with all types of system-related problems, from logging in to obtaining and installing software updates to tracking down and fixing obscure network issues.

Much of the material in this chapter applies equally to Mac OS X versions 10.3 and later, and to both Mac OS X and Mac OS X Server. Where information is specific to a particular version or release, it is so marked.

Because so many services exist, along with different ways to configure or use them, this chapter cannot discuss every system configuration or every action you will have to take as a system maintainer. Instead, it familiarizes you with the concepts you need to understand and the tools you need to use to maintain a Mac OS X system. Where it is not possible to go into depth about a subject, the chapter provides references to other sources.

This chapter assumes that you are familiar with the following terms. Refer to the Glossary (page 919) for their definitions.

block (device) | environment | mount (a device) | spawn
daemon | filesystem | process | system console
device | fork | root filesystem | X server
device filename | kernel | runlevel |
disk partition | login shell | signal |


Philosophy of System Administration

Classic UNIX system administration involves editing configuration files with a text editor such as vim. Newer UNIX-like systems include tools, some of which have a graphical user interface, to help you manipulate these configuration files. But whatever the interface, most UNIX-like system administration is based on text configuration files. As a result, if you know what you are doing, you can always work with these text files instead of using a graphical tool.

Graphical maintenance tools

From the beginning, the Macintosh has provided graphical tools to perform system administration tasks. These tools, as opposed to some of the tools found on other UNIX-like systems, are designed from the ground up to have a graphical user interface. Under Mac OS X there may be no file you can edit in place of using the graphical tool. If you are used to doing system administration by editing text files, you may find the lack of editable files frustrating.

Among other benefits, using a consistent front end for a system administration task allows Apple to change the way something works within Mac OS X while retaining the same interface. For example, although the way secure remote login services are launched changed from version 10.3 to version 10.4, the interface remained the same. You still enable Remote Login in the Sharing preferences. Disk management, checking, and repair are provided by the Disk Utility application. In general, the Macintosh favors the use of GUI tools for server administration. Configuration files, pathnames, and other details may change from one release to another, and sometimes they may differ between the regular and Server versions of the system.

System Preferences

Many of the tasks common to system maintenance on OS X systems can be handled through the System Preferences application (page 27). The Accounts pane allows you to create, delete, or modify user accounts. Servers can be started and stopped from the Sharing pane, and network configuration can be managed from the Network pane.

Server Admin and Workgroup Manager

Mac OS X Server provides two additional maintenance tools: Server Admin and Workgroup Manager. Some of the functions that appear in System Preferences on a non-Server system are found in these applications on a Server system. In general, Workgroup Manager controls resources such as shared disks or printers and user accounts, while Server Admin controls services such as DHCP, email, and Web services.

Textual maintenance tools

This chapter focuses on text-based system maintenance tools. These tools are less crucial on the Macintosh than they are on traditional UNIX systems, but they offer convenience and flexibility, and they integrate well with other UNIX utilities. The underlying principles remain the same as with the graphical tools.


System Administrator and Superuser

Much of what a system administrator does is work that ordinary users do not have permission to do. When performing one of these tasks, the system administrator logs in as root or uses sudo (page 431) to have systemwide powers that are beyond those of ordinary users: A user with root privileges is referred to as Superuser. The username is root by default. On an OS X Server system, the root account initially has the same password as the first user. On a non-Server system, the root account is disabled by default. Superuser has the following powers and more:

  • Some commands, such as those that add new users, partition hard drives, and change system configuration, can be executed only by root. Superuser can use certain tools, such as sudo, to give specific users permission to perform tasks that are normally reserved for Superuser.

  • Read, write, and execute file access and directory access permissions do not affect root. Superuser can read from, write to, and execute all files, as well as examine and work in all directories.

  • Some restrictions and safeguards that are built into some commands do not apply to root. For example, root can change any user's password without knowing the old password.

When you are running with root (Superuser) privileges, the shell by convention displays a special prompt to remind you of your status. By default this prompt is or ends with a pound sign (#). In graphical applications, such as System Preferences, root privileges are generally denoted by a lock icon. If the lock is open, you have root privileges. When you click a closed lock, the system prompts you for a username and password. When you enter the username and password for root or for an administrative user (a member of the admin group), the lock icon opens and you have root privileges.

Security: Change the root password

On an OS X Server system, the root account starts with the same password as the first account that was created on the system. It is a good idea to change this password to help prevent users from making unwanted changes to the system. Because users with administrative accounts can use sudo (page 431) to obtain root privileges, changing the root password does not offer complete protection against attacks, but it is a step in the right direction.

Caution: Least privilege

When you are working on the computer, especially when you are working as the system administrator, perform any task by using the least privilege possible. When you can perform a task logged in as an ordinary user, do so. When you must be logged in as Superuser, do as much as you can as an ordinary user, log in or use sudo so that you have root privileges, do as much of the task as has to be done as Superuser, and revert to being an ordinary user as soon as you can. Because you are more likely to make a mistake when you are rushing, this concept becomes more important when you have less time to apply it.

You can gain or grant Superuser privileges in a number of ways:

  1. When you bring the system up in single-user mode (page 437), you are Superuser.

  2. Once the system is up and running in multiuser mode (page 438), if you have enabled the root account, you can log in as root. When you supply the proper password, you will be running as Superuser.

  3. You can give an su (substitute user) command while you are logged in as yourself and, with the proper password, will have Superuser privileges. For more information refer to "su: Gives You Another User's Privileges" on page 430.

  4. You can use sudo selectively to give users Superuser privileges for a limited amount of time on a per-user and per-command basis. See page 431 for more information on sudo.

    Security: root-owned setuid programs are extremely dangerous

    Because a root-owned setuid program allows someone who does not know the root password to have the powers of Superuser, it is a tempting target for a malicious user. Your site should have as few of these programs as necessary.

  5. Any user can create a setuid (set user ID) file (page 90). Setuid programs run on behalf of the owner of the file and have all of the owner's access privileges. While you are running as Superuser, you can change the permissions of a file owned by root to setuid. When an ordinary user executes a file that is owned by root and has setuid permissions, the program has full root privileges. In other words, the program can do anything that root can do and that the program does or allows the user to do. The user's privileges do not change. When the program finishes running, all user privileges are restored to the way they were before the program was started. Setuid programs that are owned by root are extremely powerful and extremely dangerous to system security, which is why very few of them are on the system. Examples of setuid programs that are owned by root include passwd, at, and crontab. The following example shows two ways for a user to give a program setuid privileges:

    $ ls -l my*
-rwxr-xr-x   1 zach zach 17000 Mar  2 02:31 myprog
-rwxr-xr-x   1 zach zach 17000 Mar  2 02:31 myprog2
$ chmod 4755 myprog
$ chmod u+s myprog2
$ ls -l my*
-rwsr-xr-x   1 zach zach 17000 Mar  2 02:31 myprog
-rwsr-xr-x   1 zach zach 17000 Mar  2 02:31 myprog2

    The s in the owner execute position of the ls l output (page 87) indicates that the file has setuid permission. Under Mac OS X 10.3.9 and later, the setuid permission bit is ignored on shell scripts.

  6. Some programs ask you for a password when they start or when you perform a function that needs Superuser privileges. For graphical applications, you can provide the username and password of an administrative user. If you are an administrative user, the sudo utility (page 431) can give you Superuser privileges using your own password. When a program asks for a password to gain Superuser privileges, you stop running as the privileged user when you quit using the program. This setup helps keep you from remaining logged in as Superuser when you do not need or intend to be. Some techniques limit the number of ways in which you can become Superuser. PAM (page 466) controls the who, when, and how of logging in. The /etc/ttys file controls which terminals (ttys) a user can log in on as root.

Security: Do not allow root access over the Internet

If you need root access to a system over a network, use ssh (page 847) to connect to the system. The root account is not enabled by default, so you must connect as an ordinary user and then use sudo to gain root privileges. If you enable the root account, you should deny direct access to it through ssh. In the file /etc/sshd_config, add the line PermitRootLogin no.

Enabling the root Account

On an OS X Server system, the root account is enabled by default and has the same password as the first user created on the system. Initially on a non-Server system there is no password for root, so there is no way to log in as root or to use su to become root. However, you can use an sudo sh or sudo -s command to spawn a root shell (a shell with Superuser privileges).

If you are an administrative user (a member of the admin group, one with the Allow user to administer this computer checked in the Accounts preference pane), you can use the following command to enable the root account by assigning a password to it:

$ sudo passwd root
Password:
Changing password for root.
New password:
Retype new password:

Enter your password at the first prompt, then enter the new root password at the next two prompts. Once you have assigned a password to root, you must use this password when you log in as root and when you use su to become root temporarily.

System Maintenance Tools

Many tools can help you be an efficient and thorough system administrator. This section describes three powerful utilities: su, which allows you to take on the identity of another user; sudo, which performs the same basic function as su but is configurable; and kill, which, when run by root, allows you to terminate any process. See "Useful Utilities" on page 448 for brief descriptions of additional maintenance tools.

su: Gives You Another User's Privileges

The su (substitute user) utility can create a shell or execute a program with the identity and permissions of a specified user. Under OS X, a system administrator frequently uses sudo in place of su. Follow su on the command line with the name of a user; if you are root or if you know that user's password, you take on the identity of that user. When you give an su command without an argument, su defaults to Superuser so that you take on the identity of root (root must have a password [see page 430] and you have to know the root password).

To be sure that you are using the system's official version of su (and not one planted on the local system by a malicious user), specify su's absolute pathname (/bin/su) when you use it. (Of course, if someone has compromised your system enough to have you run a fake su command, you are in serious trouble anyway, but using an absolute pathname for su is still a good idea.)

Security: Superuser, PATH, and security

The fewer directories you keep in your PATH when you are root, the less likely you will be to execute an untrusted program as root. If possible, keep only the default directories, along with /sbin and /usr/sbin, in root's PATH. Never include the working directory as. or :: anywhere in PATH, or : as the first or last element of PATH. For more information refer to "PATH: Where the Shell Looks for Programs" on page 285.

When you give an su command to become Superuser, you spawn a new shell, which displays the # prompt. You return to your normal status (and your former shell and prompt) by terminating this shell: Press CONTROL-D or give an exit command. Giving an su command by itself changes your user and group IDs but makes only minimal changes to your environment. When you give the command su (you can use l in place of the hyphen), you get a root login shell: It is as though you logged in as root. Not only are your user and group IDs those of root, but your entire environment is that of root. The login shell executes the appropriate startup scripts before displaying a prompt.

Use the id utility to display the changes in your user and group IDs and in the groups you are associated with.

$ id
uid=501(alex) gid=501(alex) groups=501(alex), 81(appserveradm), ...
$ su
Password:
# id
uid=0(root) gid=0(root) groups=0(wheel), 1(daemon), 2(kmem), ...

sudo: Performs Specific Tasks as Another User

The sudo utility offers a more flexible approach to using the privileges of another user. As a member of the admin group you can modify the /etc/sudoers file to configure the privileges offered by sudo. Unlike su, sudo uses your password to authenticate, not the password of the user whose privileges you want to use. Under OS X, any account in the admin group can use sudo to run any command as root, even if the root account has not been enabled (there is no root password). Give the command su l (lowercase "l," for list) to display the list of commands you are allowed to run with sudo.

The sudo utility takes its arguments as the command it is to run. The following example first shows that a user is not permitted to kill (page 432) a process. With the use of sudo and the user's password, an administrative user is permitted to kill the process.

$ kill -15 4982
-bash: kill: (4982) - Operation not permitted
$ sudo kill -15 4982
Password:
$

You can also use sudo to obtain a root shell by giving an argument of bash or by using the s option:

$ sudo -s
Password:
#

The sudo utility enables you to execute a command as a user other than root. The following command allows Max, who must be a member of the admin group, to list the contents of Sam's home directory while avoiding running a command as root:

$ ls ~sam
ls: /Users/sam: Permission denied
$ sudo -u sam ls ~sam
src personal bin

When you have successfully run sudo within the last five minutes it does not ask for a password again. This courtesy reduces the temptation to start a root shell, possibly forgetting to exit from it.

kill: Sends a Signal to a Process

The kill builtin sends a signal to a process. This signal may or may not terminate (kill) the process, depending on which signal is sent and how the process is designed. Refer to "trap: Catches a Signal" on page 577 for a discussion of the various signals and the way a process receives them. Running kill is not the first method a user or system administrator should try when a process needs to be aborted.

Caution: kill: method of last resort

Because of its inherent dangers, using kill is a method of last resort, especially when you are working as Superuser. One kill command issued by root can bring the system down without warning.

When you do need to use kill, send the termination signal (kill TERM or kill 15) first. Only if that signal does not work should you attempt to use the kill signal (kill KILL or kill 9).

Usually a user can kill a process by working from another window or by logging in on another terminal. Sometimes you may have to log in as root (or use su) to kill a process for a user. To kill a process, you need to know its PID. The ps utility can give you this information once you know the name of the program the user is running and/or the username of the user. The top utility (page 874) can also be helpful in finding and killing a runaway process (see top's S command).

In the following example, Zach complains that Safari is stuck and that he cannot do anything from the Safari window, not even close it. A more experienced user could open another window and kill the process or use the Force Quit option on the Apple menu. In this case, you kill it for Zach. First use ps with the U option, followed by the name of the user, to view all processes associated with that user:

$ ps -U zach
  PID  TT  STAT     TIME COMMAND
  133  ??  Ss    0:03.92 /System/Library/Frameworks/ApplicationServices.framew
14191  ??  Ss    0:01.18 /System/Library/CoreServices/loginwindow.app/Contents
14203  ??  Ss    0:00.43 /System/Library/CoreServices/pbs
14210  ??  S     0:00.96 /System/Library/CoreServices/Dock.app/Contents/MacOS/
14211  ??  S     0:01.99 /System/Library/CoreServices/SystemUIServer.app/Conte
14212  ??  S     0:01.26 /System/Library/CoreServices/Finder.app/Contents/MacO
14220  ??  S     0:00.17 /usr/sbin/sshd -i
14247  ??  S     0:04.86 /Applications/Safari.app/Contents/MacOS/Safari -psn_0
14222  p1  Ss    0:00.08 -bash

This list is fairly short, and the process running Safari is easy to find. If the list is long, you can use grep with the i option (for a search that is not case sensitive) to find the process running Safari:

$ ps -U zach | grep -i safari
14247  ??  S      0:04.87 /Applications/Safari.app/Contents/MacOS/Safari -psn_0

Now that you know the PID of Zach's process running Safari is 14247, you can use kill to terminate it. The safest way to use kill is to log in as Zach (perhaps allow him to log in for you or su to zach [su zach] if you are logged in as root) and give the command

$ kill TERM 14247

Only if this command fails should you send the kill signal:

$ kill KILL 14247

The KILL option instructs kill to send a SIGKILL signal, which the process cannot ignore. You can give the same command while you are logged in as root, but a typing mistake can have much more far-reaching consequences than when you make the same mistake while you are logged in as an ordinary user. A user can kill only his own processes, whereas Superuser can kill any process, including system processes. (There are a few exceptions: You may be temporarily unable to kill a process waiting on disk activity, and the launchd or init process with PID 1 cannot be killed.)

As a compromise between speed and safety, you can combine the su and kill utilities by using the c option to su. The following command runs the part of the command line following the c with the identity of Zach:

# su zach -c "kill -TERM 14247"

killall

A useful utility related to kill is killall. It is very similar to kill but uses a command name in place of a PID number. To kill all your processes that are running iTunes or vim, you can give the following command:

$ killall iTunes vim

When given by root, this command kills all processes running iTunes or vim.

You can also list processes and send signals using the Activity Monitor application (found in the Utilities folder within Applications).

Avoiding a Trojan Horse

A Trojan horse is a program that does something destructive or disruptive to your system while appearing to be benign. As an example, you could store the following script in an executable file named ifconfig:

while true
    do
    echo 'Good Morning Mr. Jones. How are you? Ha Ha Ha.' > /dev/console
    done

If you are running as Superuser when you run this command, it would continuously write a message to the console. If the programmer were malicious, it could do worse. The only thing missing in this plot is access permissions.

A malicious user could implement this Trojan horse by changing Superuser's PATH variable to include a publicly writable directory at the start of the PATH string. (The catch is that you need to write to /etc/profilewhere the PATH variable is set for rootand only root can do that.) Then you would need to put the bogus ifconfig program file in that directory. Because the fraudulent version appears in a directory mentioned earlier than the real one in PATH, the shell runs it. The next time Superuser tries to run ifconfig, the fraudulent version would run instead of the system version.

Trojan horses that wait for and take advantage of the misspellings that most people make are among the most insidious types. For example, you might type sl instead of ls. Because you do not regularly execute a utility named sl and you may not remember typing the command sl, it is more difficult to track down this type of Trojan horse than one that takes the name of a more familiar utility.

A good way to help prevent the execution of a Trojan horse is to make sure that your PATH variable does not contain a single colon (:) at the beginning or end of the PATH string or a period (.) or double colon (::) anywhere in the PATH string. This precaution ensures that you will not execute a file in the working directory by accident. A common way to check for a possible Trojan horse is to examine the filesystem periodically for files with setuid (refer to item 5 on page 429). The following command lists these files:

# find / -perm -4000 -exec ls -lh {} \; 2> /dev/null
rwsrwsr-x    1 root  admin    30K Feb 14 03:19 /Applications/Server/AppleSha<...>
rwsrwxr-x    1 root  wheel    17K Feb 14 03:25 /Applications/System Preferen<...>
rwsrwxr-x    1 root  admin    52K Feb 14 03:13 /Applications/Utilities/Activ<...>
...
-rwsr-xr-x   1 root  wheel    31K Feb 13 21:03 /bin/ps
-r-sr-xr-x   1 root  wheel    24K Feb  7 04:05 /bin/rcp
-rwsr-xr-x   1 root  wheel    69K Feb 14 03:38 /sbin/launchd
-r-sr-xr-x   1 root  wheel    19K Feb 13 20:56 /sbin/mount_nfs
-r-sr-xr-x   1 root  wheel    322K Feb 14 03:29 /sbin/mount_smbfs
-r-sr-xr-x   1 root  wheel    32K Feb  7 04:05 /sbin/ping
-r-sr-xr-x   1 root  wheel    41K Feb  7 04:09 /sbin/ping6
-r-sr-sr-x   2 root  tty    73K Feb 13 20:57 /sbin/restore
-r-sr-xr-x   1 root  wheel    33K Feb  7 04:05 /sbin/route
-r-sr-sr-x   2 root  tty    73K Feb 13 20:57 /sbin/rrestore
...

This command uses find (page 728) to locate all files that have their setuid bit set (mode 4000). For more information refer to "Access Permissions" on page 87. The hyphen preceding the mode causes find to report on any file that has this bit set, regardless of how the other bits are set. The output sent to standard error is redirected to /dev/null (page 444) so that it does not clutter the screen.

Getting Help

Mac OS X comes with extensive documentation (page 27). Apple maintains a Web page that points to many useful support documents: www.apple.com/support. You can also find help on the System Administrator's Guild site (www.sage.org). In addition, the Internet is a rich source of information on managing a UNIX system of any sort; refer to Appendix B (page 907) and to the author's home page (www.sobell.com) for pointers to useful sites.

You do not need to act as a Mac OS X system administrator in isolation; a large community of OS X experts is willing to assist you in getting the most out of your system, although you will get better help if you have already tried to solve a problem yourself by reading the available documentation. If you are unable to solve a problem by reviewing the documentation, a well-thought-out question to the appropriate newsgroup, such as comp.sys.mac.system, or mailing list can often provide useful information. Make sure you describe the problem and identify your system carefully. Include information about your version of OS X and any software packages and hardware that might relate to the problem. For a good paper by Eric S. Raymond and Rick Moen titled "How to Ask Questions the Smart Way" see www.catb.org/~esr/faqs/smart-questions.html.


System Operation

This section covers the basics of how the system functions and how you can make intelligent decisions as a system administrator. It does not cover every aspect of system administration in the depth necessary to set up or modify all system functions. Instead, it provides a guide to bringing a system up and keeping it running from day to day.

The way that system services run changed from OS X version 10.3 to version 10.4. Version 10.3 used init for system startup and launched services (daemons) using xinetd. Some services ran constantly in the background. Under version 10.4 and later, system startup and all daemons are run by launchd. This section describes how systems running OS X 10.4 and later work.

Booting the System

Booting a system is the process of reading the Mac OS X kernel (page 939) into system memory and starting it running. Although it is unlikely that you will need to do so, you can use the Startup Disk preference pane of System Preferences to modify the boot procedure.

OS X 10.4 and later

As the last step of the boot procedure, Mac OS X 10.4 runs the launchd program as PID number 1. The launchd program is the first genuine process to run after booting and is the parent of all system processes.

OS X 10.3 and earlier

Mac OS X 10.3 loads mach_init, which in turn loads init as well as any programs found in /etc/mach_init.d. System scripts, such as /etc/rc, are run by init and start other programs, such as xinetd, which provide services. The init program is the first genuine process to run after booting and is the parent of all system processes.

Startup Scripts

The first script that launchd runs is /etc/rc, the system startup script. This script checks the disks, starts system services, loads drivers, and so on. However, it is only part of the system startup process. Exactly what this script does depends on the system configuration. For example, on a system that is netbooting, rc runs the /etc/rc.netboot script, which handles network startup. If the /etc/rc.local file exists, /etc/rc runs it.

Because Apple has announced plans to phase out the rc script, avoid making changes to /etc/rc or using /etc/rc.local. If you want to add new services, set up a new launchd job (page 456) or create a new system startup item by copying and modifying the code of an existing item in /System/Library/StartupItems. Put your item in /Library/StartupItems to avoid conflicts with system startup items.

Most system services are started by launchd. Services that run on demand or at regular intervals are run from launchd. SystemStarter starts the services found in /Library/StartupItems and /System/Library/StartupItems and that must be launched in a particular order, as well as those that are dependent on other services.

Tip: List the kernel boot messages

As the system comes up, the kernel generates messages. To save the list of kernel boot messages, give the following command immediately after booting the system and logging in:

$ dmesg > dmesg.boot

This command saves the kernel messages in the dmesg.boot file. This list can be educational. It can also be useful when you are having a problem with the boot process. For more information refer to dmesg on page 717.

No runlevels

Unlike Linux and System V UNIX systems, OS X does not have multiple runlevels. It can come up in single-user mode (where services are not started) or in multiuser mode (where services are started). Instead of a set of directories containing scripts to run when entering each runlevel, OS X uses configuration files to specify when each service should be started. Typically you do not need to modify these files.

You can enable and disable most system services from the Services tab of the Sharing preference pane. Making changes here affects the /etc/hostconfig file and certain files in the /System/Library/StartupItems, /System/Library/LaunchAgents, and /System/Library/Launch-Daemons directories. Other services are controlled through launchd's configuration files (page 456) and the files in the StartupItems directories.

Single-User Mode

You can put a Mac OS X system into single-user mode by holding down COMMAND-S while it is booting. When the system is in single-user mode, only the text console (page 35) is enabled. You can run programs from the console in single-user mode as you would from any terminal in multiuser mode. The differences are that not all filesystems may be mounted, so you may not be able to access some files, and few of the system daemons will be running. In single-user mode, the root filesystem (startup volume) is mounted readonly, so you cannot modify any files. If you need to modify files (for instance, to fix a configuration problem that prevents the system from booting normally), you must mount the required filesystem for both reading and writing.

If you need to modify a file on the root filesystem, you must remount it in read-write mode. Before you remount it, you should check it for errors. Although fsck is deprecated for most uses, Apple recommends using it to check filesystems for errors and correct errors in single-user mode:

# /sbin/fsck -fy

Once you have checked the root filesystem, you can remount it in read-write mode using the mount utility:

# /sbin/mount -uw /

The u (update) option causes mount to change the state of a mounted filesystem, not mount an unmounted filesystem. The w (write) option specifies the state to change; it mounts the filesystem in read-write mode.

With the system in single-user mode, you can perform system maintenance that requires either unmounted filesystems or just a quiet systemno one except you using it, so that no user programs interfere with disk maintenance and backup programs. The classical UNIX term for this state is quiescent.

Going Multiuser

After you have determined that all is well with all filesystems, you can bring the operating system up to multiuser mode. When you exit from the single-user shell, launchd brings the system up to multiuser mode. When it goes from single-user to multiuser mode, the system runs the /etc/rc script as described in "Startup Scripts" on page 436.

Multiuser Mode

Multiuser mode is the normal state for a Mac OS X system. All appropriate filesystems are mounted and users can log in from all connected terminals, dial-up lines, and network connections. All support services and daemons are enabled and running. Once the system is in multiuser mode, you will see a login screen or prompt. By default OS X systems are set up to boot directly to multiuser mode without stopping at single-user mode. By default non-Server OS X systems skip the login screen and log in the first user account automatically.

Logging In

The system launches the graphical login window by defaultyou have to request the text console if you want a textual login. For more information refer to "More About Logging In" on page 34 and "Logging In on the Text Console" on page 35.

OS X uses PAM (page 466), the Pluggable Authentication Module facility, which gives you a great deal of control over user logins and authentication. Using PAM, you can specify multiple levels of authentication, mutually exclusive authentication methods, or parallel methods, each each of which is sufficient to grant access to the system. For example, you can have a different authentication method for console logins and for network logins. You can require that modem users authenticate themselves via two or more methods (such as a smartcard or badge reader and a password). PAM modules also provide security technology vendors with a convenient way to integrate their hardware or software products into a system.

The default PAM configuration on Mac OS X uses NetInfo (page 441) for account and password verification. All passwords in the NetInfo database are encrypted or hashed. It is not feasible to recover an encrypted password. When you log in, the login process encrypts/hashes the password you type at the prompt and compares it to the encrypted/hashed password in the database. If it matches, you are authenticated.

Textual login

With a textual login, launchd starts a getty process, which displays a login: prompt. When you enter your username, getty establishes the characteristics of your terminal. It then overlays itself with a login process and passes to the login process whatever you entered in response to the login: prompt. Next login prompts you for a password and verifies the password you enter. If either your username or your password is not correct, login displays Login incorrect and prompts you to log in again.

Graphical login

With a graphical login, the launchd process spawns loginwindow on the console, providing features similar to getty and login. The loginwindow utility presents a login window, then uses PAM to authenticate the user. If the user is authenticated, the loginwindow program brings up the Finder and any programs configured to be started automatically upon login. These programs are controlled through the Login Items tab of the System Preferences Accounts pane.

Whether you are using login or loginwindow, the program handling your login assigns values to the HOME, PATH, LOGNAME, SHELL, and TERM variables. It looks in the NetInfo group database to identify the groups you belong to. When the program has finished its work, it overlays itself with the login shell (for a textual login) or the Finder (for a graphical login), which inherits these variables.

During a textual login, the login shell assigns values to additional shell variables and executes the commands in the system startup shell scripts /etc/profile and /etc/bashrc (or, if you are using tcsh, /etc/login and /etc/cshrc).

After executing the system startup commands, the shell executes the commands from the personal startup shell scripts in your home directory (page 259). Because the shell executes these scripts after the system scripts, a sophisticated user can override any variables or conventions that were established by the system, whereas a new user can remain uninvolved in these complications.

Periodic Tasks

OS X performs many system maintenance functions periodically. These tasks are handled by the /etc/daily, /etc/weekly, and /etc/monthly shell scripts. The daily script is run at 3:15 every morning, the weekly script at 3:15 on Saturday mornings, and the monthly script at 5:30 on the morning of the first day of each month. These scripts are launched by launchd under version 10.4 and later, and by cron on earlier systems.

Each of these scripts checks for a corresponding local script file containing additional maintenance tasks and runs these scripts using /bin/sh. The local files have .local filename extensions. For example, /etc/daily checks for a script named /etc/daily.local.

Running a Program and Logging Out

When OS X displays a shell prompt, you can execute a program or exit from the shell. If you exit from a shell, the process running the shell dies and the parent process wakes up. When the shell is a child of another shell, the parent shell wakes up and displays a prompt. Exiting from a login shell causes the operating system to send launchd a signal that one of its children has died. Upon receiving this signal, launchd takes action based on the contents of the /etc/ttys file. In the case of a process controlling a line for a terminal, for example, launchd informs getty that the line is free for another user.

When you are using a graphical login and exit from the GUI, launchd starts a new loginwindow session to allow a new login.

Bringing the System Down

The shutdown and halt utilities perform the tasks needed to bring the system down safely. These utilities can restart the system, prepare the system to be turned off, put the system in single-user mode, and, on some hardware, power down the system. The reboot utility is linked to halt. The shutdown utility calls halt to actually perform the system shutdown, but only after sending messages to any users logged in on textual devices. A user on the graphical console is not warned by the system and must be warned manually.

Caution: shutdown may not save your work

Unlike the Shut Down option on the Apple menu, the shutdown and halt utilities do not check with applications to see whether it is safe to shut the system down. If you are running a GUI, use the Shutdown option on the Apple menu to bring the system down.

You must tell shutdown when you would like to bring the system down. The time can be expressed as an absolute time of day, as in 19:15, which causes the shutdown to occur at 7:15 P.M. Alternatively, you can give the number of minutes from the present time, as in +15, which means 15 minutes from now. To bring the system down immediately (recommended for emergency shutdowns only or when you are the only user logged in), you can give the argument +0, or its synonym: now. Non-root logins are disabled 5 minutes before shutdown, or immediately if shutdown is sooner than that.

Calling shutdown with the r option causes the system to reboot (same as the reboot command except that reboot implies now). Using h instead of r forces the system to halt (same as the halt command except that halt implies now). A message appears once the system has been safely halted: System halted. Normally, however, the system will power off, preventing you from seeing the message.

Because Mac OS X is a multiuser system, shutdown warns textual users before taking any action. This warning gives users a chance to prepare for the shutdown, perhaps by writing out editor files or exiting from networking applications. You can replace the default shutdown message with one of your own by following the time specification on the command line with a message:

# /sbin/shutdown -h 09:30 Going down 9:30 to install disk, up by 10:00 am.

Caution: Do not turn the power off before bringing the system down

Avoid rebooting a system without first bringing it down as described here. OS X, like other UNIX and UNIX-like systems, speeds up disk access by keeping an in-memory collection of disk buffers that are written to the disk periodically or when system use is momentarily low. When you turn off or reset the computer without writing the contents of these disk buffers to the disk, you lose the information held in the buffers. Running shutdown forces these buffers to be written. You can force the buffers to be written at any time by issuing a sync command. However, sync does not unmount filesystems, nor does it bring the system down.

Going Single-User

The only way to put an OS X system in single-user mode is to boot or reboot into it. You cannot go directly from multiuser to single-user mode as you can on some UNIX systems. For more information refer to "Single-User Mode" on page 437.

Turning the Power Off

Shutting the system down is quite straightforward: Give the command halt to bring the system down. This is equivalent to selecting Shut Down from the Apple menu. Running halt requires root privileges.

Crash

A crash occurs when the system suddenly stops or fails when you do not intend it to do so. A crash may result from software or hardware problems or from a loss of power. As a running system loses power, nothing is regular or predictable. In a fraction of a second, some components are supplied with enough voltage; others are not. Buffers are not flushed, corrupt data may be written to the hard disk, and so on. IDE drives do not behave as predictably as SCSI drives under these circumstances. After a crash, you must bring the operating system up carefully to minimize possible damage to the filesystems. Frequently there will be little or no damage, and the system will recover without any help beyond pressing the power button.

On a Macintosh, a crash frequently indicates a hardware problem. Third-party utilities are available for testing and diagnosing hardware problems. One such tool is TechTool Pro (www.micromat.com).

Repairing a Filesystem

Although the filesystems are checked automatically during the boot process if needed, you will have to check them manually if a problem cannot be repaired automatically. To check the filesystems manually after a crash, use the Disk Utility application or the diskutil command line utility (both perform the same function).

If a file is corrupt or missing, you will have to restore it from a backup.

When the System Does Not Boot

When you cannot boot the computer from the hard drive, you may need to rerun the Mac OS X installer from the disks that came with the computer. There are several ways to reinstall the operating system. Be careful to choose the technique that does not erase your data.


NetInfo

The NetInfo system is a hierarchical database that stores user and system information that is frequently shared among systems on a local network. It is similar in scope and function to NIS (page 945). NetInfo information is divided into domains, with searches beginning in the local domain and proceeding to higher-level domains as needed. This setup allows local servers to handle requests for local resources and to hand off requests to other servers if necessary. In a default installation, the local domain (.) and the root domain (/) are the same.

Each domain comprises nodes, each of which has one or more properties, and some of which hold other nodes. A property is a name/value pair that allows the system to look up attributes such as the username or UID corresponding to a user account. As in the OS X filesystem, nodes that hold other nodes are called directories. Table 11-1 lists the standard top-level directories for a domain.

Table 11-1. NetInfo top-level directories

Directory

Contents

aliases

Mail aliases, such as mailing lists

groups

Groups

machines

Host information, such as IP addresses

mounts

Filesystem mount information

networks

A list of known networks (now replaced by DNS)

printers

Available printers and their characteristics

protocols

Protocols, such as IP or UDP

rpcs

RPC service numbers

services

Internet service numbers, such as 80 for HTTP

users

System user database, including home directories and passwords

The following example uses nireport to list the names of all top-level directories in the root domain:

$ nireport / / name
users
groups
machines
networks
...

The nireport utility takes at least three arguments. The first argument is the name of the domain to search, typically / (the root domain). The second argument specifies a path to search within that domain. The / path is the top level, analogous to the root of the UNIX filesystem. Using /users as a second argument would cause nireport to search the user accounts. The third and following arguments specify the properties for which you want nireport to display information.

The users top-level directory contains one subdirectory for each user account. Each of these subdirectories includes a name/username property and a uid/UID property. The next command displays the values associated with the name and uid names of each directory in the /users directory of the root domain:

$ nireport / /users name uid
nobody  -2
root    0
daemon  1
...
zach    501
sam     502
max     503

On a typical Mac OS X installation, you can browse the NetInfo database using the NetInfo Manager application (found in the Utilities folder within Applications). If you have administrator privileges, you can use NetInfo Manager to modify the NetInfo database. Under Mac OS X Server, the Server Admin and Workgroup Manager applications also work with the NetInfo databases. Table 11-2 lists the command line utilities that work with the NetInfo database.

Table 11-2. Command line utilities that work with NetInfo

Utility

Function

nidump

Dumps NetInfo databases as text (see CMDREF)

nifind

Finds a given directory in the NetInfo hierarchy

nigrep

Lists directories containing items matching a regular expression

nireport

Displays fields from entries in a NetInfo database

niload

Reads a flat file database into NetInfo

niutil

Manipulates and displays a NetInfo database

The nidump utility displays all of the properties and subdirectories of a node in the database. The following example displays the properties for Max's account:

$ nidump -r /users/max /
{
"name" = ( "max" );
"_writers_passwd" = ( "max" );
"_writers_tim_password" = ( "max" );
"_writers_picture" = ( "max" );
"shell" = ( "/bin/bash" );
"_writers_hint" = ( "max" );
"home" = ( "/Users/max" );
"gid" = ( "503" );
"authentication_authority" = ( ";ShadowHash;" );
"_writers_realname" = ( "max" );
"picture" = ( "/Library/User Pictures/Animals/Dog.tif" );
"passwd" = ( "********" );
"realname" = ( "Max" );
"hint" = ( "" );
"sharedDir" = ( "Public" );
"_shadow_passwd" = ( "" );
"uid" = ( "503" );
"generateduid" = ( "C7A16427-E9B9-11D9-83B0-000D935C83CE" );
}

The NetInfo database is substantially more flexible than traditional UNIX flat files, and it can be extended as needed. For more information see the netinfo man page in section 5 of the system manual.


Important Standard Directories and Files

Files on a Mac OS X system are kept in standard locations in the filesystem hierarchy. These locations reflect a combination of POSIX standards, BSD traditions, and Apple's additions. For example, the /Applications directory is an Apple addition. Some directories are documented in the man page for hier, which describes part of the Mac OS X filesystem hierarchy.

~/.bash_profile

Startup file Holds a user's login shell initialization script. The Bourne Again Shell executes the commands in this file in the same environment as the shell each time a user logs in. The file must be located in a user's home directory.

You can use .bash_profile to specify a terminal type (for vim, terminal emulators, and other programs), run stty to establish the terminal characteristics that a user desires, and perform other housekeeping functions when a user logs in. For more information refer to "Startup Files" on page 259.

~/.bashrc

Startup file Holds a user's interactive, nonlogin shell initialization script. The Bourne Again Shell executes the commands in this file in the same environment as the (new) shell each time a user creates a new interactive shell. The .bashrc script differs from .bash_profile in that it is executed each time a new shell is spawned, not just when a user logs in. For more information refer to "Startup Files" on page 259.

~/Desktop

Personal Desktop The subdirectories in this directory hold files that appear on a user's Desktop in the Finder.

~/Library

Personal libraries and support Holds application support files for a user, similar to those stored in /Library and /System/Library. It also holds some of a user's startup (preference) files.

/

Root The ancestor of all files in the OS X filesystem. The startup disk (page 87) is mounted as the root directory.

/Applications

Application software Holds graphical application programs, including both standard system applications and applications that you install. The /Applications directory contains subdirectories that hold groups of related applications, such as the system tools in the Utilities subdirectory and the server maintenance tools in the Server subdirectory on Mac OS X Server.

/bin

Essential command binaries Holds the files needed to boot the system and run it when it first comes up in single-user mode (page 437).

/dev

Device files Holds the files that represent peripheral devices, such as disk drives, terminals, and printers (page 100). Some peripheral devices, such as mice and USB printers, are not represented by files.

/dev/null

Null file Output sent to this file disappears; also called a bit bucket. The /dev/null file is a device file that must be created with mknod. Input that you redirect to come from this file appears as nulls, creating an empty file. You can create an empty file named nothing by giving the command

$ cat /dev/null > nothing

or

$ cp /dev/null nothing

or without explicitly using /dev/null

$ > nothing

The last command redirects the output of a null command to the file with the same result as the previous commands. You can use this technique to truncate an existing file to zero length without changing its permissions. You can also use /dev/null to get rid of output that you do not want:

$ grep portable * 2> /dev/null

The preceding command looks for the word portable in all files in the working directory. Any output to standard error (page 262), such as permission or directory errors, is discarded; output to standard output appears on the screen.

Optional

/dev/zero

Zero file Input you take from this file contains an infinite string of zeros (numerical zeros, not ASCII zeros). The dd (page 703) utility converts and copies a file; you can fill a file or overwrite a file with zeros with a command such as the following. The od utility (page 799) displays the contents of the new file:

$ dd if=/dev/zero of=zeros bs=1024 count=10
10+0 records in
10+0 records out
10240 bytes transferred in 0.000301 secs (34006075 bytes/sec)
$ ls -l zeros
-rw-rw-r--  1 sam  sam  10240 Dec  3 20:26 zeros
$ od -c zeros
0000000 \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 \0
*
0024000

When you try to do with /dev/zero what you can do with /dev/null, you quickly fill the partition you are working in:

$ cp /dev/zero bigzero
cp: bigzero: No space left on device
$ rm bigzero

/etc

Machine-local system configuration Holds administrative, configuration, and other system files. This file is a symbolic link to /private/etc.

Caution: Files in /etc are not always used

On traditional UNIX systems, the user account database and other similar databases are stored in files in /etc. Under Mac OS X, these databases are stored in NetInfo (page 441) and access to them is handled through the lookupd daemon (page 455). See Table VI-18 on page 796 for a list of NetInfo databases. The files holding the traditional UNIX databases may exist and contain some valid information under OS X. However, making modifications to these files has no effect. For example, user accounts are not listed in /etc/passwd, even though it contains a list of system accounts such as root. To list user accounts give the command nidump passwd /. Refer to page 796 for more information on nidump.

/etc/motd

Message of the day Displayed each time someone performs a textual login. This file typically contains site policy and legal information. Keep this file short because users tend to see the message many times.

/etc/pam.d

PAM files Holds files that specify authentication methods used by programs that use PAM (page 466).

Caution: Be cautious when changing PAM files

Unless you understand how to configure PAM, avoid changing the files in /etc/pam.d. Mistakes in the configuration of PAM can make the system unusable. For more information refer to "PAM" on page 466.

/etc/printcap

Printer capability database Derived from 4.3BSD UNIX, describes system printers and gives utilities such as lpr access to the Mac OS X printing environment. The system's printer configuration application automatically maintains this file.

/etc/profile

Startup file Contains a systemwide interactive shell initialization script. When you log in using bash, the shell does immediately execute the commands in this file in the same environment as the shell. (For more information on executing a shell script in this manner, refer to the discussion of the . [dot] command on page 261.) This file allows the system administrator to establish systemwide environment parameters that individual users can override. Using this file, the system administrator can set shell variables, execute utilities, and take care of other housekeeping tasks. See also ~/.bash_profile on page 444.

See the /etc/profile file on the local system for a more complex example.

/etc/protocols

Protocols Provides protocol numbers, aliases, and brief definitions for DARPA Internet TCP/IP protocols. Do not modify this file.

/etc/rpc

RPC services Maps RPC services to RPC numbers. The three columns in this file show the name of the server for the RPC program, the RPC program number, and aliases.

/etc/services

System services Lists system services. The three columns in this file show the informal name of the service, the port number/protocol the service frequently uses, and aliases for the service. This file does not specify which services are running on the local system, nor is it used to map services to port numbers. The services file is used internally to map port numbers to services for display purposes.

/etc/X11

Machine-local configuration for the X Window System

/Library

Third-party libraries and support Holds libraries and support files that are not part of the base system. Contrast with /System/Library.

/private

System-local files Holds files that cannot be shared between machines. The /etc, /tmp, and /var files are symbolic links to the /private/etc, /private/tmp, and /private/var directories, respectively. This setup allows systems to share the root filesystem (startup volume) over a network while retaining individual copies of core system files.

/private/etc

The target of the /etc symbolic link.

/private/tmp

The target of the /tmp symbolic link.

/private/var

The target of the /var symbolic link.

/sbin

Essential system binaries Utilities used for system administration are stored in /sbin and /usr/sbin. The /sbin directory includes utilities needed during the booting process, and /usr/sbin holds utilities used after the system is up and running. Programs that were in /etc on older UNIX systems will generally be in one of these directories.

/System

System files Holds system software and support code, including standard system fonts, kernel extensions, and the preference files that control launchd.

/System/Library

System libraries and support files Holds libraries and support files that are part of the base system and that are used with the Carbon and Cocoa APIs (page 478). This directory serves a function similar to that of /usr/lib.

/tmp

Temporary files Holds temporary files for many programs. This directory is a symbolic link to /private/tmp.

/Users

User home directories Holds users' home directories. It is possiblealbeit not commonto change the location of home directories.

/usr

Second major hierarchy Traditionally includes subdirectories that contain information used by the system. Files in /usr subdirectories do not change often and may be shared by several systems.

/usr/bin

Most user commands Contains the standard Mac OS X utility programsthat is, binaries that are not needed in single-user mode (page 437).

/usr/include

Header files included by C programs

/usr/lib

Libraries Holds static and shared libraries that are used with the UNIX API (page 478). Contrast with /System/Library.

/usr/sbin

Nonvital system administration binaries See /sbin.

/usr/share

Architecture-independent data Subdirectories of /usr/share include dict, doc, info, locale, man, misc, terminfo, and zoneinfo.

/usr/share/doc

Miscellaneous documentation

/usr/share/file/magic

Magic number database Most files begin with a unique identifier called a magic number. This file is a text database listing all known magic numbers on the system. When you use the file utility, it consults /usr/share/file/magic to determine the type of a file. Occasionally you will acquire a new tool that creates a new type of file that is unrecognized by the file utility. In this case you need to update the /usr/share/file/magic file; refer to the magic man page for details. See also "magic number" on page 941.

/usr/share/info

GNU info system's primary directory

/usr/X11R6

X Window System, version 11, release 6 Holds subdirectories that contain X11 binaries, libraries, and support files.

/var

Variable data Files with contents that vary as the system runs are found in subdirectories under /var. The most common examples are temporary files, system log files, spooled files, and user mailbox files. Subdirectories can include backups, log, run, spool, tmp, and yp. Older versions of UNIX scattered such files through several subdirectories of /usr (/usr/adm, /usr/mail, /usr/spool, /usr/tmp). This directory is a symbolic link to /private/var.

/var/log

Log files Holds lastlog (a record of the last login by each user), system.log (system messages from syslogd), and wtmp (a record of all logins/logouts).

/var/root

Home directory for root

/var/spool

Spooled application data Holds cups, mqueue, postfix, and samba.

/Volumes

Automounted volumes Holds automatically mounted volumes (filesystems). For more information refer to "/Volumes" on page 86.


Useful Utilities

This section briefly describes a few of the many utilities that can help you perform system administration tasks. Read the man/info pages for these utilities to learn more about using them. Some of these utilities are useful to users other than the system administrator.

chsh

Changes the login shell for a user. When you call chsh without an argument, you change your own login shell. Superuser can change the shell for any user by calling chsh with that user's username as an argument. When changing a login shell with chsh, you must specify an installed shell that is listed in /etc/shells. OS X rejects entries that are not in this file. Also, you must give the pathname to the shell exactly as it appears in /etc/shells. In the following example, Superuser changes Max's shell to tcsh:

#chsh -s /bin/tcsh max
chsh: netinfo domain "." updated

The s option allows you to specify the pathname of the new shell. If you do not specify a new shell, chsh opens an editor showing the account features you are allowed to change. Change the pathname adjacent to Shell:, write the file to disk, and exit from the editor to change the login shell.

clear

Clears the screen. You can also use CONTROL-L from bash to clear the screen. The value of the environment variable TERM (page 913) determines how to clear the screen.

diskutil

Formats and partitions disks (page 712).

dmesg

Displays recent system log messages (page 717).

hdiutil

Manipulates disk images; comparable to diskutil.

launchctl

Sends instructions to launchd. You can use this utility to start system services or change their settings (page 765).

newfs newfs_hfs newfs_msdos

Creates a new filesystem. Each of these utilities builds a a different type of filesystem. The newfs utility builds a UFS filesystem and works on either a hard disk partition or a floppy diskette. The following command builds a UFS filesystem on the device at /dev/disk0s3:

# newfs /dev/dsk0s3

Substitute the pathname of the device you want to create the filesystem on for /dev/disk0s3. Use newfs_hfs to create an HFS filesystem and newfs_msdos to create an MS-DOS filesystem. The following command creates an MS-DOS filesystem on the device at /dev/disk1s7:

# newfs_msdos /dev/disk1s7

Although these utilities accept various options and arguments, none are required. The diskutil utility (page 712) offers a different interface to the same functionality.

ping

Sends packets to a remote system. This utility determines whether you can reach a remote system through the network and the time it takes to exchange messages with the remote system. Refer to "ping: Tests a Network Connection" on page 405.

reset (link to tset)

Resets terminal characteristics. The value of the environment variable TERM (page 913) determines how to reset the screen. The kill and interrupt characters are set to their default values and character echo is turned on. The reset utility is useful for restoring the screen to a sane state after it has been corrupted. Similar to an stty sane command.

sysctl

Displays and alters kernel variables. You can alter some kernel options and limits by modifying variables using this utility. For more information see page 858.

umask

A shell builtin that specifies the mask that the system uses to set access permissions when you create a file. See umask on page 883 for more information.

uname

Displays information about the system. Without any arguments, uname displays the name of the operating system (Darwin). Giving uname a a (all) option causes it to display the operating system name, the hostname, the version number and release date of the operating system, and other information:

# uname a
Darwin localhost 8.0.0 Darwin Kernel Version 8.0.0: Sun Feb 13 18:22:16 PST 2005; root:xnu-
756.2.obj~2/RELEASE_PPC Power Macintosh powerpc

rpcinfo

The rpcinfo utility can display information about programs registered with portmap and can make RPC calls to programs to see if they are alive. For more information on portmap, refer to "RPC Network Services" on page 417. The rpcinfo utility takes the following options and arguments:

rpcinfo p [host]
rpcinfo [n port] u \ t host program [version]
rpcinfo b \ d program version

p

probe Lists all RPC programs registered with portmap on host or on the local system if host is not specified.

n

(port) number With t or u, uses port port in place of the port number specified by portmap.

u

UDP Makes a UDP RPC call to version (if specified) of program on host and reports whether there was a response.

t

TCP Makes a TCP RPC call to version (if specified) of program on host and reports whether there was a response.

b

broadcast Makes an RPC broadcast to version of program and lists hosts that respond.

d

delete Removes local RPC registration for version of program. Can be used by Superuser only.

Give the following command to see which RPC programs are registered with the portmap daemon on the system named peach:

$ /usr/sbin/rpcinfo -p peach
   program vers proto   port
    100000   2    tcp    111  portmapper
    100000   2    udp    111  portmapper
    100024   1    udp  32768  status
    100024   1    tcp  32768  status
    100021   1    udp  32769  nlockmgr
    100021   3    udp  32769  nlockmgr
...

Use the u option to display a list of versions of a daemon, such as ypserv, registered on a remote system (peach):

$ /usr/sbin/rpcinfo -u peach ypserv
program 100004 version 1 ready and waiting
program 100004 version 2 ready and waiting

Specify localhost to display a list of versions of a daemon registered on the local system:

$ /usr/sbin/rpcinfo -u localhost nfs
program 100003 version 2 ready and waiting
program 100003 version 3 ready and waiting

The portmap daemon is not needed for Apple's native file-sharing protocols. It is started automatically if you enable a service that uses it, such as NFS. Otherwise, it is not started.

Tip: Set the clocks

The portmap daemon relies on the client's and the server's clocks being synchronized. A simple DoS attack (page 931) can be initiated by setting the server's clock to the wrong time.


Setting Up a Server

This section discusses issues that are commonly encountered when setting up servers: rules for writing traditional configuration files and ways to specify hosts and subnets. This section also covers the OS X property list files and the interactions between the configuration files used by specific daemons and the configuration files used by the operating system. Also in this section are discussions of the launchd superserver, strategies to secure a server under OS X, and DHCP.

Configuration Files

Services are controlled by property list files (page 453) in /System/Library/LaunchDaemons and traditional UNIX configuration files. Examples of traditional configuration files include Apache's /etc/httpd/httpd.conf and sshd's /etc/sshd_config. To configure services manually, you need to understand both types of configuration files.

Traditional Configuration File Conventions

Most traditional configuration files, typically named *.conf, rely on the following conventions:

  • Blank lines are ignored.

  • A # anywhere on a line starts a comment that continues to the end of the line. Comments are ignored.

  • When a name contains a SPACE, you must quote the SPACE by preceding it with a backslash (\) or by enclosing the entire name within single or double quotation marks.

  • To make long lines easier to read and edit, you can break them into several shorter lines. Break a line by inserting a backslash (\) immediately followed by a NEWLINE (press RETURN in a text editor). When you insert the NEWLINE before or after a SPACE, you can indent the following line to make it easier to read. Do not break lines in this manner while editing on a Windows machine, as the NEWLINEs may not be properly escaped (Windows uses RETURN-LINEFEEDs to end lines).

Configuration files that do not follow these conventions are noted in the text.

Specifying Clients

Table 11-3 shows some of the common ways to specify a host or a subnet. Most of the time you can specify multiple hosts or subnets by separating the host or subnet specifications with SPACEs. Different programs accept different forms; check the documentation for the program you are working with.

Table 11-3. Specifying a client

Client name pattern

Matches

n.n.n.n

One IP address.

name

One hostname, either local or remote.

name that starts with .

Matches a hostname that ends with the specified string. For example, .example.com matches the systems kudos.example.com and speedy.example.com, among others.

IP address that ends with .

Matches a host address that starts with the specified numbers. For example, 192.168.0. matches 192.168.0.0 192.168.0.255. If you omit the trailing period, this format does not work.

starts with @

Specifies a netgroup.

n.n.n.n/m.m.m.m or n.n.n.n/mm

An IP address and subnet mask specify a subnet.

starts with /

An absolute pathname of a file containing one or more names or addresses as specified in this table.

Wildcards

* and ?

Matches one (?) or more (*) characters in a simple hostname or IP address. These wildcards do not match periods in a domain name.

ALL

Always matches.

LOCAL

Matches any hostname that does not contain a period.

Operator

EXCEPT

Matches anything in the preceding list that is not in the following list. For example, a b c d EXCEPT c matches a, b, and d. Thus you could use 192.168. EXCEPT 192.168.0.1 to match all IP addresses that start with 192.168. except 192.168.0.1.

Examples

Each of the following examples specifies one or more systems:

10.10. | Matches all systems with IP addresses that start with 10.10.
.apple.com | Matches all named hosts on the Apple network.
localhost | Matches the local system.
127.0.0.1 | The loopback address; always resolves to the local host.
192.168.*.1 | Could match all routers on a network of /24 subnets.

Specifying a Subnet

When you set up a server, you frequently need to specify the clients that are allowed to connect to that server. Sometimes it is convenient to specify a range of IP addresses, called a subnet. The discussion on page 397 explains what a subnet is and how you can use a subnet mask to specify a subnet. Usually, you can specify a subnet as

n.n.n.n/m.m.m.m

or

n.n.n.n/mbits

where n.n.n.n is the base IP address and the subnet is represented by m.m.m.m (the subnet mask) or mbits (the number of bits used for the subnet mask). For example, 192.168.0.1/255.255.255.0 represents the same subnet as 192.168.0.1/24. In binary, decimal 255.255.255.0 is represented by 24 ones followed by 8 zeros. The /24 is shorthand for a subnet mask with 24 ones. Each line in Table 11-4 presents two notations for the same subnet followed by the range of IP addresses that the subnet includes.

Table 11-4. Different ways to represent a subnet

Bits

Mask

Range

10.0.0.0/8

10.0.0.0/255.0.0.0

10.0.0.0 10.255.255.255

172.16.0.0/12

172.16.0.0/255.240.0.0

172.16.0.0 172.31.255.255

192.168.0.0/16

192.168.0.0/255.255.0.0

192.168.0.0 192.168.255.255

Property List Files

A property list is a structured data file, usually with a filename extension of .plist, that stores settings or configuration information. Each element in a property list is a key/value pair. A list of key/value pairs is called a dictionary. A key is typically a string, and a value is either a string, binary data, an array of values, or another (embedded) dictionary. The launchd.plist man page provides examples of the elements of a property list.

Mac OS X stores a property list in one of three formats: plain text, XML, or binary. The application or utility that accesses the property list does not need to know which format the list is inthe operating system provides the application with the information in a format it can use. Many system defaults are implemented through property lists, so even applications that do not access property lists directly are likely to use them indirectly.

The plutil utility (page 815) checks the syntax of property list files and converts them from one format to another.

Plain-Text Property Lists

The original property list format was a human-readable text file, but this format may be phased out in future releases of Mac OS X. A plain-text property list key is a string that must be quoted in the rare case where it contain SPACEs. A string value is represented as a quoted string, an array as a comma-separated list of quoted strings surrounded by parentheses, and a dictionary as a list of name/value pairs enclosed between braces. Following is an example of a plain-text property list:

$ cat /System/Library/StartupItems/Samba/StartupPreferences.plist
{
  Description     = "SMB File Service";
  Provides        = ("SMB File Service");
  Requires        = ("Disks", "DirectoryServices");
  OrderPreference = "None";
  Messages =
  {
    start = "Starting SMB File Service";
    stop  = "Stopping SMB File Service";
  };
}

The Description key is associated with a simple string value. The Provides and Requires keys are each associated with an array, although Provides has only one element in its array. The Messages key is associated with a dictionary that contains two name/value pairs.

The plutil utility does not generate plain-text property lists.

XML Property List Files

Many property list files are stored in XML (page 962) format. This format, which is the most commonly used under OS X, is human readable, can be edited by hand, and allows easy and reliable syntax checking.

XML uses tags to identify elements in a file. XML tags start with an opening angle bracket (<) and end with a closing angle bracket (>). Opening tags comprise a word surrounded by angle brackets (such as <key>). Closing tags generally comprise the same word, preceded with a slash (/) and surrounded by angle brackets (such as </key>).

The next example uses plutil to convert the plain-text property list shown in the preceding example to XML format and cat to display the converted file:

$ plutil -convert xml1 StartupParameters.plist
$ cat StartupParameters.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
           <key>Description</key>
           <string>SMB File Service</string>
           <key>Messages</key>
           <dict>
                   <key>start</key>
                   <string>Starting SMB File Service</string>
                   <key>stop</key>
                   <string>Stopping SMB File Service</string>
           </dict>
           <key>OrderPreference</key>
           <string>None</string>
           <key>Provides</key>
           <array>
                   <string>SMB File Service</string>
           </array>
           <key>Requires</key>
           <array>
                   <string>Disks</string>
                   <string>DirectoryServices</string>
           </array>
</dict>
</plist>

The XML format is more verbose, but clearer than the plain-text format. It also supports additional data types, such as integer and Boolean, as shown in the following XML property list fragment:

<key>Display</key>
<integer>0</integer>
<key>DockSwitch</key>
<true/>

Some XML elements do not have opening and closing tags. These elements use a trailing slash to indicate that they have just one tag. The <true/> element in the preceding example demonstrates such a case.

Binary Property List Files

The binary property list file format is the most space efficient, but is not human readable. If you need to edit a binary property list file, you must convert it to XML, edit the XML, and then convert it back to binary, as in the following example:

$ plutil -convert xml1 -o copy.plist original.plist
$ vim copy.plist
[...]
$ plutil -convert binary1 -o original.plist copy.plist

lookupd: Which Service to Look at First

In addition to NetInfo (page 441), Mac OS X supports a number of other ways to search for information about users and hosts. With the advent of NIS and DNS, it was no longer a simple matter of searching a local file for user and system information. Whereas you once looked in /etc/passwd to get user information and looked in /etc/hosts to find system address information, you can now use several methods to find this type of information. The lookupd daemon provides a centralized source for lookups from DNS, NetInfo, text files on the local disk, and directory services. Directory services, in turn, provide a way to access other directory systems, such as LDAP (page 940).

The /etc/lookupd directory holds files that override defaults. For example, if a users file exists in this directory, it changes the way that lookupd obtains user records. Alternatively, lookupd can get configuration information from NetInfo in the corresponding NetInfo directories /config/lookupd or /locations/lookupd. For more information refer to the lookupd man page.

The Superserver

The superserver is a daemon that launches, on demand, other daemons that provide services. Starting service daemons on demand, as opposed to allowing them to run constantly, conserves system resources. The superserver under OS X version 10.4 and later is launchd and under version 10.3 and earlier is xinetd.

The launchd Superserver

Mac OS X 10.4

The launchd daemon, which was introduced in Mac OS X 10.4, replaces several core system daemons from earlier versions, including xinetd, init, and cron. This section focuses on the launchd functionality that replaces xinetd.

The launchd daemon is responsible for the services that have property lists in the /System/Library/LaunchDaemons and /Library/LaunchDaemons directories. Each program launchd controls is called a job. A job is defined by an XML property list (configuration) file and identified by a unique label, denoted by the Label key, in that file. Once it loads (reads) a configuration file, launchd controls the job defined by that file and launches the associated program as needed. The launchd daemon can launch a program at system startup, periodically, or on demand.

The launchd daemon controls only programs it has started; it does not control programs started in other ways. The following property list configuration file, which is associated with the sshd daemon, shows some of the keys used by launchd:

$ cat /System/Library/LaunchDaemons/ssh.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>Label</key>
        <string>com.openssh.sshd</string>
        <key>Program</key>
        <string>/usr/libexec/sshd-keygen-wrapper</string>
        <key>ProgramArguments</key>
        <array>
                 <string>/usr/sbin/sshd</string>
                 <string>-i</string>
        </array>
        <key>SessionCreate</key>
        <true/>
         <key>Sockets</key>
         <dict>
                  <key>Listeners</key>
                  <dict>
                         <key>Bonjour</key>
                         <array>
                                <string>ssh</string>
                                <string>sftp-ssh</string>
                         </array>
                         <key>SockServiceName</key>
                         <string>ssh</string>
                  </dict>
          </dict>
          <key>StandardErrorPath</key>
          <string>/dev/null</string>
          <key>inetdCompatibility</key>
          <dict>
                  <key>Wait</key>
                  <false/>
          </dict>
</dict>
</plist>

Only the Label and ProgramArguments keys are required. The Label key identifies the job. You can use the Label key as an argument to launchctl (next) to start, stop, load, or unload a job. Give the command launchctl list to display the Label key of each loaded job.

The ProgramArguments key provides launchd with the pathname of the program to run and the arguments to call it with. The preceding example instructs launchd to execute /usr/sbin/sshd with the i option, which indicates that sshd is being launched by another daemon.

Internet servers are generally run on demand, in response to incoming connections. The Sockets key, if present, indicates that launchd should listen on the specified port and launch the program whenever it detects an incoming connection. In the preceding example, the sshd daemon is run whenever a connection is received on the port associated with the ssh service, as defined in /etc/services. The Bonjour key is used with the Bonjour system for automatic service discovery.

You can use the Mac OS X graphical system administration utilities to start and stop jobs. Typically you do not need to edit the launchd configuration files.

Daemons obtain configuration information from their own configuration files. For example, the sshd daemon uses /etc/sshd_config, a standard UNIX configuration file (page 451).

The jobs specified by the property lists in the /System/Library/LaunchDaemons and /Library/LaunchDaemons directories are loaded when the system boots. The jobs specified by the property lists in /System/Library/LaunchAgents, /Library/LaunchAgents, and ~/Library/LaunchAgents are loaded when a user logs in. These directories are discussed in "Important Standard Directories and Files" on page 444.

launchctl: Controls launchd

The launchd daemon runs all the time. You can change the behavior of launchd while it is running by using launchctl (page 765). The first argument to launchctl is called a subcommand. For example, the list subcommand requests a list of all jobs that launchd controls. When you give a launchctl list command as a nonroot user, launchctl lists jobs controlled by your private instance of launchd. The following example shows typical output of this command when given by root:

# launchctl list
com.apple.KernelEventAgent
com.apple.mDNSResponder
com.apple.nibindd
com.apple.periodic-daily
com.apple.periodic-monthly
com.apple.periodic-weekly
com.apple.portmap
com.apple.servermgrd
com.apple.syslogd
com.apple.watchdogtimerd
com.vix.cron
com.apple.ntalkd
org.postfix.master
org.xinetd.xinetd
com.openssh.sshd
com.apple.ls
com.apple.atprintd.i9900
com.apple.cups-lpd
com.apple.fingerd

The load subcommand of launchctl causes launchd to read a job's configuration file(s) and start the job as specified (at system startup [immediately if the system is running], periodically, or on demand). However, if the configuration file specifies a value of <true/> for the Disabled key, launchd does not load the job. The unload subcommand tells launchd to unload the configuration files and not to run the job. You must load an unloaded job if you want to run it.

The stop subcommand of launchctl causes launchd to stop running a (loaded) job but not to unload it. Thereafter, a periodic or demand-driven job may start up again from a stopped state. A start subcommand starts a loaded job that has been stopped but has no effect on a job that is not loaded.

When you modify the configuration file of a loaded job, you must instruct launchd to read the new values from the file. Because a load subcommand has no effect on a loaded job, you must first unload the job and then load it again when you change a configuration file.

Job Requirements

The launchd daemon monitors the programs it starts. If a program terminates quickly, launchd restarts it; if it keeps terminating, however, launchd stops the job until an administrator fixes the problem and restarts the job. Many traditional UNIX daemons detach themselves from their calling process. This behavior causes launchd to believe that the daemon has failed and is prohibited in programs started by launchd.

Because launchd provides resource management and security features in a standardized way, programs run by launchd should not handle their own resource management or security. For instance, a daemon being started by launchd should specify the user and group it will run as using the UserName and GroupName keys, respectively, in its property list file rather than setting them from some other configuration file.

In some cases, such as when a daemon needs to bind to a privileged port (page 414), a daemon may need to initially run with root privileges and subsequently drop these privileges. Such a daemon would be launched once, using the RunAtLoad key, rather than run on demand.

Multiple Instances

There is always at least one launchd daemon running on a Mac OS X system, and there may be more. Nonroot users can start their own copies of launchd to run jobs and services they submit. If you use launchctl to start a new service and you do not have root privileges, the job will be attached to a new instance of launchd running with your privileges.

The xinetd Superserver

The xinetd daemon, which is used under OS X 10.3 and earlier, is a more secure replacement for the inetd superserver that originally shipped with 4.3BSD. The Internet superserver listens for network connections. When one is made, it launches a specified server daemon and forwards the data from the socket (page 101) to the daemon's standard input. The xinetd superserver performs these tasks and provides additional features to control access.

The version of xinetd distributed with Mac OS X is built with TCP wrappers support, so it uses the /etc/hosts.allow and /etc/hosts.deny files for access control (see the hosts_access man page for more information). Using TCP wrappers can simplify configuration but hides some of the more advanced features of xinetd.

The base configuration for xinetd is stored in the /etc/xinetd.conf file. Additional services are configured through files in /etc/xinetd.d.

The following xinetd configuration file allows telnet connections from the local system and any system with an IP address that starts with 192.168. This configuration file does not rely on TCP wrappers, so it does not depend on the hosts.allow and hosts.deny files.

$ cat /etc/xinetd.d/telnet
service telnet
{
       socket_type     = stream
       wait            = no
       user            = root
       server          = /usr/sbin/in.telnetd
       only_from       = 192.168.0.0/16 127.0.0.1
       disable         = no
}

The socket_type indicates whether the socket uses TCP or UDP. TCP-based protocols establish a connection between the client and the server and are identified by the type stream. UDP-based protocols rely on the transmission of individual datagrams and are identified by the type dgram.

When wait is set to no, xinetd handles multiple, concurrent connections to this service. Setting wait to yes causes xinetd to wait for the server process to complete before handling the next request for that service. In general, UDP services should set wait to yes and TCP services should set wait to no. If you were to set wait to yes for a service such as telnet, only one person would be able to use the service at a time.

The user specifies the user that the server runs as. If the user is a member of multiple groups, you can also specify the group on a separate line using the keyword group. The user directive is ignored if xinetd is not run as root. The server provides the pathname of the server program that xinetd runs for this service.

The only_from specifies which systems xinetd allows to use the service. Use IP addresses only because using hostnames can render the service unavailable if DNS fails. Zeros at the right of an IP address are treated as wildcards; thus 192.168.0.0 allows access from any system in the 192.168 subnet.

The disable line can disable a service without removing the configuration file. For more information see the xinetd man page.

Securing a Server Under Mac OS X 10.4 and Later

When you open a local system to access from other systems you must ensure that you:

  • Open the local system only to systems you want to allow to access it.

  • Allow each remote system to access only the data you want it to access.

  • Allow each remote system to access data only in the manner you want it to (read only, read/write, write only).

You can restrict which remote systems have access to a given service by using the operating system's builtin firewall. Under Mac OS X you can configure the firewall using System Preferences; under Mac OS X Server you need to use Server Admin.

You can also restrict a local server's access to the local system. A server can be run with resource limits to keep it from consuming too much memory or creating files that consume too much disk space, it can be run with the privileges of an ordinary user, and it can be restricted to part of the filesystem. Every limitation you impose on a server makes it more difficult for a potential security flaw in that server to affect the rest of the system.

The launchd daemon provides control over all of these restrictions.

Setting Up Resource Limits

Resource limits can control the amount of memory and CPU time as well as the disk space used by a process. You can set both hard and soft resource limits in a launchd configuration file. Soft limits are enforced, but a program can request that the limits be raised. Soft resource limits cannot be raised above the hard resource limits, however, and hard resource limits cannot be raised at all. For example, the following resource limits from a property list file prevent a daemon from consuming more than five seconds of CPU time. (On a multitasking system, it will likely take more than five seconds of real time before a single process has used five seconds of CPU time.)

<key>HardResourceLimits</key>
<dict>
         <key>CPU</key>
         <integer>5</integer>
         <key>Data</key>
         <integer>33554432</integer>
         <key>ResidentSetSize</key>
         <integer>33554432</integer>
         <key>NumberOfFiles</key>
         <integer>16</integer>
         <key>Stack</key>
         <integer>33554432</integer>
         <key>NumberOfProcesses</key>
         <integer>96</integer>
         <key>FileSize</key>
         <integer>33554432</integer>
</dict>

If you specify any HardResourceLimits, any limits that you do not specify are implicitly set to 0, which will normally prevent a job from running. The NumberOfProcesses limit is not merely the number of processes that are part of a job, but includes all processes owned by the user. An idle graphical console might have 50 processes running. To limit a job's resources without impeding a user's normal tasks, it is best to create a new account with limited privileges that runs the job and does nothing else.

The NumberOfProcesses limit used in the preceding example is high. The default limit under Mac OS X is 100 processes per user, with a systemwide limit of 532 processes in total. You can alter these values using sysctl (page 858). The default limits are much higher (1,000 and 2,048 respectively) under Mac OS X Server.

Setting Up a chroot Jail

On early UNIX systems, the root directory was a fixed point in the filesystem. On modern UNIX variants, including Mac OS X, you can define the root directory on a per-process basis. The chroot utility allows you to run a process with a root directory other than /.

The root directory is the top of the directory hierarchy and has no parents: A process cannot access files above the root directory (because they do not exist). If, for example, you run a program (process) and specify its root directory as /Users/sam/jail, the program would have no concept of any files in /Users/sam or above: jail is the program's root directory and is labeled / (not jail).

By creating an artificial root directory, frequently called a (chroot) jail, you prevent a program from being able to access and modifypossibly maliciouslyfiles outside the directory hierarchy starting at its root. You must set up a chroot jail properly to increase security: If you do not set up a chroot jail correctly, you can actually make it easier for a malicious user to gain access to a system than if there were no chroot jail.

Using chroot

Creating a chroot jail is simple: Working as root, give the command /usr/sbin/chroot directory. The directory becomes the root directory and the process attempts to run the default shell. Working as root from the /Users/sam directory, the following example attempts to set up a chroot jail in the (existing) /Users/sam/jail directory:

# /usr/sbin/chroot jail
chroot: /bin/sh: No such file or directory

In this example, chroot sets up the chroot jail, but when it attempts to run the sh shell, it fails. Once the jail is set up, the directory that was named jail takes on the name of the root directory: / and chroot cannot find the file identified by the pathname /bin/sh. The chroot jail is working perfectly but is not very useful.

Getting a chroot jail to work the way you want is more complicated. To have the preceding example run bash in a chroot jail, you need to create a bin directory in jail (/Users/sam/jail/bin) and copy /bin/sh to this directory. Because the bash binary (and the copy of bash named sh) is dynamically linked to shared libraries (page 486), you need to copy these libraries into jail as well. The libraries go in /lib. The following example creates the necessary directories, copies bash, uses otool to display the shared library dependencies of bash, and copies the necessary libraries, and the dyld dynamic linker to load them, into lib:

$ pwd
/Users/sam/jail
$ mkdir -p bin usr/lib
$ cp /bin/sh bin
$ otool -L bin/bash
bin/sh:
        /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.0.0)
$ cp /usr/lib/libncurses.5.4.dylib usr/lib
$ cp /usr/lib/libSystem.B.dylib usr/lib
$ cp /usr/lib/dyld usr/lib

Now that everything is set up, you can try starting the chroot jail again. All of the setup can be done by an ordinary user but you have to run chroot as Superuser:

$ su
Password:
# /usr/sbin/chroot .
dyld: Library not loaded: /usr/lib/system/libmathCommon.A.dylib
  Referenced from: /usr/lib/libSystem.B.dylib
  Reason: image not found
Trace/BPT trap
#

This attempt is closer, but it turns out that the libSystem dynamic library has additional dependencies. Creating the usr/lib/system subdirectory and copying libmathCommon into it solves the problem:

# mkdir usr/lib/system
# cp /usr/lib/system/libmathCommon.A.dylib usr/lib/system
# chroot .
sh-2.05b# pwd
/
sh-2.05b# ls
sh: ls: command not found

This time chroot finds and starts sh, which displays its default prompt (sh-2.05b#). The pwd command works because it is a shell builtin (page 138). However, bash cannot find the ls utility (it is not in the chroot jail). You can copy /bin/ls into the jail if you want users in the jail to be able to use ls.

To set up a useful chroot jail, you need to determine which utilities the users of the chroot jail will need. Then you can copy the necessary binaries and their libraries into the jail.

For jobs using launchd's InetdCompatibility mode, you need the program /usr/libexec/launchproxy in the jail. This program provides an environment similar to that provided by inetd or xinetd.

Because users are looked up dynamically from a database, you do not need any password files or authentication programs in the jail unless the program you are using accesses these files or programs directly.

Tip: Keeping multiple chroot jails

If you plan to deploy multiple chroot jails, it is a good idea to keep a clean copy of the files in bin, usr/lib, and usr/lib/system somewhere other than in one of the active jails.

Running a Service in a chroot Jail

Running a shell inside a jail has limited use. You are more likely to need to run a specific service inside the jail. To do so, you need make sure all the files needed by that service are inside the jail. Once you have set up a functional jail, you can specify the jail as the RootDirectory of the service in the launchd configuration file.

Several servers are set up to take advantage of chroot jails. For example, the FTP server (ftpd) can automatically start chroot jails for clients; see the ftpd.conf man page.

Security Considerations

Some services need to be run as root, but they relinquish their root privileges once started (Procmail and tnftpd are examples). If you are running such a service, you do not need to specify a user for it in the launchd configuration file.

It is possible for a process run as root to escape from a chroot jail. For this reason, you should always run a service as another user before starting a program running inside the jail. Also, be careful about which setuid (page 90) binaries you allow inside a jail: A security hole in one of these can compromise the security of the jail. Also, make sure the user cannot access executable files that she uploads.

Running Without root Privileges

Some services can run with restricted privileges, rather than with full root privileges. Services should never be run with root privileges unless it is absolutely necessary. The most common reasons for a server to need root privileges are to run in a chroot jail or to listen on a privileged port (page 414). A service started from launchd does not need root privileges for these functions; because launchd runs with root privileges, it can listen on a privileged port, run the program inside a chroot jail, and still run the program as a nonroot user. To specify that a program runs with a specific user's privileges, give the user's name in the UserName key. Group privileges can be specified with the GroupName or InitGroups keys.

DHCP

Instead of having network configuration information stored in local files on each system, DHCP (Dynamic Host Configuration Protocol) enables client systems to retrieve network configuration information each time they connect to the network. A DHCP server assigns temporary IP addresses from a pool of addresses to clients as needed.

This technique has several advantages over storing network configuration information in local files:

  • DHCP enables a new user to set up an Internet connection without having to work with IP addresses, netmasks, DNS addresses, and other technical details. DHCP also makes it faster for an experienced user to set up a connection.

  • DHCP facilitates assignment and management of IP addresses and related network information by centralizing the process on a server. A system administrator can configure new systems, including laptops that connect to the network from different locations, to use DHCP. DHCP then assigns IP addresses only when each system connects to the network. The pool of IP addresses is managed as a group on the DHCP server.

  • DHCP enables IP addresses to be used by more than one system, reducing the total number of IP addresses needed. This conservation of addresses is important given that the Internet is quickly running out of IPv4 addresses. Although one IP address can be used by only one system at a time, many end-user systems require addresses only occasionally, when they connect to the Internet. By reusing IP addresses, DHCP lengthens the life of the IPv4 protocol. DHCP applies to IPv4 only, as IPv6 forces systems to configure their IP addresses automatically (called autoconfiguration) when they connect to a network (page 400).

DHCP is particularly useful for administrators who are responsible for a large number of systems because it removes the requirement for individual systems to store unique configuration information. DHCP allows an administrator to set up a master system and deploy new systems with a copy of the master's hard disk. In educational establishments and other open-access facilities, it is common to store the hard disk image on a shared drive and have each workstation automatically restore itself to pristine condition at the end of each day.

You can obtain more information from the DHCP Web site (www.dhcp.org) and from the DHCP FAQ (www.dhcp-handbook.com/dhcp_faq.html).

How DHCP Works

The client system contacts the server daemon (bootpd on a Macintosh) to obtain the IP address, netmask, broadcast address, nameserver address, and other networking parameters. The server, in turn, provides a lease on the IP address to the client. The client can request specific terms of the lease, including its duration, and the server can limit these terms. While connected to the network, a client typically requests extensions of its lease as necessary so its IP address remains the same. The lease can expire once the client is disconnected from the network, or the server can give the client a new IP address when it requests a new lease. You can also set up a DHCP server to provide static IP addresses for specific clients (refer to "Static Versus Dynamic IP Addresses" on page 394).

DHCP is broadcast based, so the client and server must be on the same subnet (page 397).

DHCP Client

A DHCP client requests network configuration parameters from the DHCP server and uses those parameters to configure the client's network interface. Under Mac OS X, DHCP setup is controlled by a setting in the Network preference pane (Figure 11-1). To enable DHCP, select the interface that you want to use DHCP for from the Network preference pane and click the TCP/IP tab. Using DHCP is the default setting.

Figure 11-1. The Network preference pane

[View full size image]

If the local network requires a DHCP client ID (a name used to identify your computer to the server), enter it in the appropriate box. Most networks do not require a client ID.

If you run into configuration problems, try clicking Renew DHCP Lease. Clicking this button restarts the DHCP negotiation.

If no DHCP server is available, the client assigns itself an address in the 169.254.0.0169.254.255.255 block. The system does not assign itself an address when no network cable is plugged in.

DHCP Server

The DHCP server maintains a list of IP addresses and other configuration parameters. When requested, the DHCP server provides configuration parameters to a client. DHCP service is supported under Mac OS X Server only and should be configured using the Server Admin application. The bootpd server can provide DHCP, BOOTP, and NetBoot services, all of which are used to allow network booting of client systems. See the bootpd man page for more information.


PAM

PAM (actually Linux-PAM, or Linux Pluggable Authentication Modules) allows a system administrator to determine how various applications use authentication (page 922) to verify the identity of a user. PAM provides shared libraries (page 486) of modules (located in /usr/lib/pam) that, when called by an application, authenticate a user. Pluggable refers to the ease with which you can add and remove modules from the authentication stack. The configuration files kept in the /etc/pam.d directory determine the method of authentication and contain a list, or stack, of calls to the modules. PAM may also use other files when necessary. Under OS X, the default PAM configuration uses NetInfo.

Instead of building the authentication code into each application, PAM provides shared libraries that keep the authentication code separate from the application code. The techniques of authenticating users stay the same from application to application. PAM enables a system administrator to change the authentication mechanism for a given application without touching the application.

PAM provides authentication for a variety of system-entry services (e.g., login, ftp). You can take advantage of PAM's ability to stack authentication modules to integrate system-entry services with different authentication mechanisms, such as RSA, DCE, Kerberos, and smart cards.

From login through the use of su to system shutdown, whenever you are asked for a password (or not asked for a password because the system trusts that you are who you say you are), PAM makes it possible for systems administrators to configure the authentication process and makes the configuration process essentially the same for all applications that use PAM to handle their authentication.

The configuration files stored in /etc/pam.d describe the authentication procedure for each application. These files usually have names that are the same as or similar to the name of the application that they configure. For example, authentication for the login utility is configured in /etc/pam.d/login. The name of the file is the name of the PAM service[1] that the file configures. Occasionally one file may serve two programs. PAM accepts only lowercase letters in the names of files in the /etc/pam.d directory.

[1] There is no relationship between PAM services and the /etc/services file. The name of the PAM service is an arbitrary string that each application gives to PAM; PAM then looks up the configuration file with that name and uses it to control how it does authentication. There is no central registry of PAM service names.

Tip: Do not lock yourself out of the system

Editing PAM configuration files correctly takes care and attention. It is easy to lock yourself out of the computer with a single mistake. To avoid this type of problem, always keep backup copies of the PAM configuration files you edit, test every change thoroughly, and make sure you can still log in once the change is installed. Keep a Superuser session open until you have finished your testing. When a change fails and you cannot log in, use the Superuser session to replace the newly edited files with their backup copies.

PAM warns you about any errors it encounters, logging them to the /var/log/secure.log file. Look in this file if you are trying to figure out why a changed PAM file is not working properly. To prevent possibly giving unnecessary information to a malicious user, PAM sends error messages to a file rather than to the screen.

For more information on PAM see the Linux-PAM System Administrators' Guide (www.kernel.org/pub/linux/libs/pam/Linux-PAM-html/pam.html). Although the URL contains the word "Linux," the documentation also applies to Mac OS X.

Configuration Files, Module Types, and Control Flags

Following is an example of a PAM configuration file. Comment lines begin with a pound sign (#).

Login module

$ cat /etc/pam.d/login
# login: auth account password session
auth        required       pam_nologin.so
auth        sufficient     pam_securityserver.so
auth        sufficient     pam_unix.so
auth        required       pam_deny.so
account     required       pam_permit.so
password    required       pam_deny.so
session     required       pam_uwtmp.so

The first line is a comment. The rest of the lines tell PAM to do something as part of the authentication process. The first word on each line is a module type indicator: account, auth, password, or session (Table 11-5). The second word is a control flag (Table 11-6) that indicates what type of action to take if authentication fails. The rest of the line contains the pathname of a PAM module and any arguments for that module. The PAM library itself uses the /etc/pam.d files to determine which modules to delegate work to.

Table 11-5. Module type indicators

Module type

Description

Controls

account

Account management

Determining whether an already authenticated user is allowed to use the service she is trying to use. (That is, has the account expired? Is the user allowed to use this service at this time of day?)

auth

Authentication

Proving that the user is authorized to use the service by using passwords or another mechanism.

password

Password changes

Updating authentication mechanisms such as user passwords.

session

Session management

Setting things up when the service is started (for example, when the user logs in) and breaking them down when the service is terminated (for example, when the user logs out).

You can use one of the control flag keywords listed in Table 11-6 to set the control flags.

Table 11-6. Control flags

Flag

Action

required

Success is required for authentication to succeed. Control and a failure result are returned after all modules in the stack have been executed. The technique of delaying the report to the calling program until all modules have been executed may keep attackers from knowing what caused their authentication attempts to fail and tell them less about the system, making it more difficult for them to break in.

requisite

Success is required for authentication to succeed. Further module processing is aborted and control is returned immediately after a module fails. This technique may expose information about the system to an attacker. Of course, if it prevents a user from giving a password over an insecure connection, it might keep information out of the hands of an attacker.

sufficient

Success indicates that this module type has succeeded, and no subsequent required modules of this type are executed. Failure is not fatal to the stack of this module type. This technique is generally used when one form or another of authentication is good enough: If one fails, PAM tries the other. For example, when you use rsh to connect to another computer, pam_rhosts first checks whether your connection can be trusted without a password. If the connection can be trusted, the pam_rhosts module reports success and PAM immediately reports success to the rsh daemon that called it. You will not be asked for a password. If your connection is not considered trustworthy, PAM starts the authentication over, asking for a password. If this second authentication succeeds, PAM ignores the fact that the pam_rhosts module reported failure. If both modules fail, you will not be able to log in.

optional

The result is generally ignored. An optional module is relevant only when it is the only module on the stack for a particular service.

PAM uses each of the module types as requested by the application. That is, the application will ask PAM separately to authenticate the user, check account status, manage sessions, and change the password. PAM will use one or more modules from the /usr/lib/pam directory to accomplish each of these tasks.

The configuration files in /etc/pam.d list the set of modules to be used for each application to perform each task. Each set of modules is called a stack. PAM calls the modules one at a time in order, from the top of the stack (the first module listed in the configuration file) to the bottom. The modules report success or failure back to PAM. When all the stacks of modules (there are exceptions) within a configuration file have been called, the PAM library reports success or failure back to the application.

Example

Part of the login service's authentication stack follows:

$ cat /etc/pam.d/login
# login: auth account password session
auth       required       pam_nologin.so
auth       sufficient     pam_securityserver.so
auth       sufficient     pam_unix.so
...

The login utility first asks for a username and then asks PAM to run this stack to authenticate the user. Refer to Table 11-5 and Table 11-6.

  1. The pam_nologin module makes sure that if the /etc/nologin file exists, only the root user is allowed to log in. (That is, the pam_nologin module reports success only if /etc/nologin does not exist or if the root user is logging in.) Thus, when a shutdown has been scheduled in the near future, the system administrator can keep users from logging in on the system only to experience a shutdown moments later.

  2. The pam_securityserver module checks with the security daemon, securityd. (On Mac OS X 10.3 and earlier, this program was named SecurityServer.) This check normally authenticates a user. Because this check is marked as sufficient, if it succeeds in authenticating the user, PAM is done and skips the next step.

  3. If necessary, the pam_unix module checks for local UNIX-style passwords. Normally this check will fail, because Mac OS X does not use the UNIX-style passwd database.

The account module type works like the auth module type but is called after the user has been authenticated. It is an additional security check or requirement for a user to gain access to the system. For example, account modules can enforce a requirement that a user can log in only during business hours.

The session module type sets up and tears down the session (perhaps mounting and unmounting the user's home directory). One common session module on a Mac OS X system is the pam_uwtmp module, which records user logins in the utmp and wtmp databases.

The password module type is a bit unusual: All modules in the stack are called once and told to get all information they need to store the password to persistent memory, such as a disk, but not actually to store it. If it determines that it cannot or should not store the password, a module reports failure. If all password modules in the stack report success, they are called a second time and told to store to persistent memory the password they obtained on the first pass. The password module is responsible for updating the authentication information (that is, changing the user's password).

Any one module can act as more than one module type. Indeed many modules can act as all four module types.

Caution: Brackets in the control flags field

You can set the control flags in a more complex way than described in the text. When you see brackets ([ ]) in the control flags position in a PAM configuration file, the newer, more complex method is in use. Each comma-delimited argument is a value=action pair. When the return from the function matches value, action is evaluated. Refer to the Linux-PAM System Administrator's Guide for more information.

Modifying the PAM Configuration

Mac OS X systems require that a user be a member of the wheel or admin group to use su. Some users might prefer not to have this restriction. PAM allows you to remove this restriction by editing the /etc/pam.d/su file:

$ cat /etc/pam.d/su
# su: auth account session
auth       sufficient     pam_rootok.so
auth       required       pam_wheel.so use_uid group=admin group=wheel
auth       sufficient     pam_securityserver.so
auth       sufficient     pam_unix.so
auth       required       pam_deny.so
account    required       pam_permit.so
password   required       pam_netinfo.so
session    required       pam_permit.so

The second line of the su module requires users to pass the checks in pam_wheel.so to become root. Commenting this line out or removing it allows users who are not members of the wheel or admin groups to use su to become root.


fink: Downloads and Installs Software

A great deal of software can be downloaded from the Internet for free. The tools discussed in Chapter 12 (page 477) cover most of what you need to know to build programs that you download from the Internet. Most software packages come with some instructions for building and installing them. Sites such as SourceForge (sourceforge.net), versiontracker (www.versiontracker.com/osx), and freshmeat (osx.freshmeat.net) can help you find thousands of software packages that can be built on Mac OS X.

Caution: Do not create /etc/pam.conf

You may have encountered PAM on other systems where all configuration is arranged in a single file (/etc/pam.conf). This file does not exist on Mac OS X systems. Instead, the /etc/pam.d directory contains individual configuration files, one for each application that uses PAM. This setup makes it easy to install and uninstall applications that use PAM without having to modify the /etc/pam.conf file each time. If you create a /etc/pam.conf file on a system that does not use this file, your PAM configuration may become confused. Do not use PAM documentation from a different system.

The fink utility (fink.sourceforge.net) automates the downloading and installation of software packages including files that a package is dependent on. The Fink project maintains a database of thousands of UNIX programs that it has ported to OS X. While fink is not distributed with Mac OS X, it is easy to download and install. The fink installer is distributed as a .dmg file containing the installation package as a standard Macintosh package file. Double-click the package to run the installer (you will need to supply an administrative password).

To avoid clashes with other packages, fink installs files in the /sw directory hierarchy. Executables are stored in /sw/bin and the installer attempts to add this directory to your PATH.

Once fink is installed, you can use it to install other packages automatically. Give the command fink --help to display a list of the functions that fink provides. The most commonly used commands are apropos, list, describe, install, and update.

The apropos command lists packages with a given word in their descriptions; the list command lists packages with a given word in their package names. For example, fink apropos web lists 32 packages but fink list web lists only the web2png package.

The following example shows fink installing the netpbm graphics package and its dependencies:

$ fink install netpbm-bin
Password:
Information about 1766 packages read in 0 seconds.
The following package will be installed or updated:
 netpbm-bin
The following 4 additional packages will be installed:
 libjpeg-shlibs libpng3-shlibs libtiff-shlibs netpbm10-shlibs
Do you want to continue? [Y/n]

Like sudo, fink uses your password (you must be a member of the admin group)not the root passwordto gain the Superuser privileges that it needs to install software.


Chapter Summary

A system administrator is someone who keeps the system useful and convenient for its users. To maintain a Mac OS X system, you must work with a variety of files and programs. Much of the work you do as the system administrator requires you to use sudo or to log in as root. The root user, called Superuser, has extensive systemwide powers that normal users do not have. Superuser can read from and write to any file and can execute programs that ordinary users are not permitted to execute.

When you bring up the system in single-user mode, only the text console is functional and not all the filesystems are mounted. When the system is in single-user mode, you can back up files and use fsck or diskutil to check the integrity of filesystems before you mount them. You can then bring the system into multiuser mode. With the system running in multiuser mode, you can still perform many administration tasks, such as adding users and printers.

System operation includes numerous functions: booting up, running startup scripts, entering single-user mode, entering multiuser mode, bringing the system down, and knowing what to do when the system crashes. The system administrator must also be familiar with NetInfo, a hierarchical database that stores user and system information that is frequently shared among systems on a local network. NetInfo is similar in scope and function to NIS.

A Mac OS X filesystem contains many important directories. They include /usr/bin, which stores most of the Mac OS X utilities, and /dev, which stores device files, many of which represent a physical piece of hardware.

Setting up a server requires an understanding of configuration files, including property lists, and lookupd, which determines what a system uses as the source of certain informationNetInfo, DNS, local files, or a combinationand in what order. The launchd superserver is a daemon that launches other daemons that provide services. A chroot jail enables you to secure a server by defining the root directory on a per-process basis, thereby limiting access to the filesystem. DHCP enables client systems to retrieve network configuration information each time they connect to the network, leading to greater system efficiency and better network resource management.

Another tool useful to system administrators is Linux-PAM, which allows you to maintain fine-grained control over who can access the system, how they can access it, and what they can do. The fink program can install third-party software packages automatically.


Exercises

1. | How does single-user mode differ from multiuser mode?
2. | How does the use of lock icons that require password authentication to open reflect the principle of least privilege?
3. | What do the letters of the su command stand for? (Hint: It is not Superuser.) What can you do with su besides give yourself Superuser privileges? How would you log in as Alex if you did not know his password but knew the root password? How would you establish the same environment that Alex has when he first logs in?
4. | How would you allow a user to execute privileged commands without giving the user the Superuser password?
5. | Assume you are working as Superuser. How do you kill process 1648? How do you kill all processes running Safari?
6. | What is the difference between LaunchAgents and LaunchDaemons? Why does each user's home directory have a Library/LaunchAgents directory but not a Library/LaunchDaemons directory?
7. | Develop a strategy for coming up with a password that an intruder would not be likely to guess but that you will be able to remember.


Advanced Exercises

8. | On Mac OS X Server, the root account starts out with the same password as the first user account created. Why is this less of a security flaw than it might seem to be?
9. | Why are setuid shell scripts inherently unsafe?
10. | If you use launchctl to load a job when you do not have root privileges, a new copy of launchd is started. Why does Mac OS X not simply load your job into the copy of launchd that is already running as process 1?
11. | Some versions of Mac OS X contain a vulnerability that allows a local user to gain root privileges. Explain how this kind of vulnerability negates the value of a chroot jail.


Part V: Programming Tools

CHAPTER 12 PROGRAMMING TOOLS

CHAPTER 13 PROGRAMMING THE BOURNE AGAIN SHELL

CHAPTER 14 THE awk PATTERN PROCESSING LANGUAGE

CHAPTER 15 THE sed EDITOR


Chapter 12. Programming Tools

IN THIS CHAPTER

Programming in C | 478
Using Shared Libraries | 486
make: Keeps a Set of Programs Current | 489
Debugging C Programs | 496
Threads | 506
System Calls | 506
CVS: Concurrent Versions System | 510

With its rich set of languages and development tools, the Mac OS X operating system provides an outstanding environment for programming. C is one of the most popular system programming languages to use in conjunction with OS X, in part because the operating system itself is written mostly in C. Using C, programmers can easily access system services using function libraries and system calls. In addition, a variety of helpful tools can facilitate the development and maintenance of programs.

This chapter explains how to compile and link C programs. It introduces the GNU gdb debugger and tools that provide feedback about memory, disk, and CPU resources. It also covers some of the most useful software development tools: the make utility and CVS (Concurrent Versions System). The make utility helps you keep track of which program modules have been updated and helps to ensure that you use the latest versions of all program modules when you compile a program. CVS is a source code management system that tracks the versions of files involved in a project.

This chapter covers the traditional UNIX command line programming environment, using utilities such as make, gcc, and cvs. Apple also provides the Xcode graphical interactive development environment (IDE), which is partially based on some of the utilities described in this chapter.

Developer Tools package (Xcode Tools)

The Developer Tools package is optional; you must install it from the distribution CD-ROMs or from the package located in /Applications/Installers. Starting with the introduction of Xcode in version 10.3, the Developer Tools package was renamed Xcode Tools, but is still referred to generically as developer tools. Check Apple's Web site (developer.apple.com) periodically for newer versions of this package.


Carbon, Cocoa, and UNIX APIs

API

An API (application program interface) is an interface used by a program to access services or functionality. APIs allow many programs to share basic functionality without duplicating development effort. Mac OS X provides several APIs for software development.

BSD API

The UNIX API is compatible with the API used for software development on other UNIX and UNIX-like systems. OS X documentation refers to this API as the BSD API, because it follows the conventions associated with BSD UNIX more often than System V UNIX, and because it offers some BSD extensions. This chapter focuses on development using the UNIX API.

Carbon API Cocoa API

The Carbon API, which has been in use since Mac OS 8, provides compatibility with pre-UNIX Mac OS systems (OS 9 and earlier). The Cocoa API, which is based on the OpenStep API, is used for native Mac OS X development.

The UNIX API is primarily targeted at C development, although it is also accessible from C++ and Objective-C. The Carbon API is used with C and C++. The Cocoa API is used with Objective-C. Cocoa classes frequently have names starting with NS, which stands for NeXT and Sun.

The Apple developer pages (developer.apple.com) have additional information about Macintosh development and APIs. This site holds Carbon documentation (developer.apple.com/carbon), Cocoa documentation (developer.apple.com/cocoa), and UNIX documentation (developer.apple.com/unix).


Programming in C

A major reason that the Mac OS X system provides an excellent C programming environment is that C programs can easily access the services of the operating system. System callsthe routines that make operating system services available to programmerscan be called from C programs. These system calls provide such services as creating files, reading from and writing to files, collecting information about files, and sending signals to processes. When you write a C program, you can use system calls in the same way you use ordinary C program modules, or functions, that you have written. For more information refer to "System Calls" on page 506.

Several libraries of functions have been developed to support programming in C. The libraries are collections of related functions that you can use just as you use your own functions and the system calls. Many of the library functions access basic operating system services through the system calls, providing the services in ways that are more suited to typical programming tasks. Other library functions, such as the math library functions, serve special purposes. Some higher-level application libraries are written in Objective-C, an object-oriented language derived from C.

This chapter describes the processes of writing and compiling C programs. However, it will not teach you to program in C.

Checking Your Compiler

The C compiler in common use on Mac OS X is GNU gcc, which comes as part of the Developer Tools package (www.gnu.org/software/gcc/gcc.html). Give the following command to determine whether you have access to the gcc compiler:

$ gcc --version
bash: gcc: command not found

If you get a response other than version information, either the compiler is not installed or your PATH variable does not contain the necessary pathname (usually gcc is installed in /usr/bin). If you get version information from the gcc command, the GNU C compiler is installed.

Next make sure that the compiler is functioning. As a simple test, create a file named Makefile with the following lines. The line that starts with gcc must be indented by using a TAB, not SPACEs.

$ cat Makefile
morning: morning.c
TAB gcc -o morning morning.c

Now create a source file named morning.c with the following lines:

$ cat morning.c
#include <stdio.h>
int main(int argc, char** argv) {
    printf("Good Morning\n");
    return 0;
}

Compile the file with the command make morning. When it compiles successfully, the resulting file will be executable; you can run the program by giving the command ./morning. When you get output from this program, you know that you have a working C compiler.

$ make morning
gcc -o morning morning.c
$ ./morning
Good Morning

A C Programming Example

You must use an editor, such as emacs or vim, to create or modify a C program. The name of the C program file must end in .c. Entering the source code for a program is similar to typing a memo or shell script. Although emacs and vim "know" that you are editing a C program, many editors do not know whether your file is a C program, a shell script, or an ordinary text document. You are responsible for making the contents of the file syntactically suitable for the C compiler to process.

Figure 12-1 illustrates the structure of a simple C program named tabs.c. The first two lines of the program are comments that describe what the program does. The string /* identifies the beginning of the comment, and the string */ identifies the end of the comment; the C compiler ignores all characters between them. Because a comment can span two or more lines, the */ at the end of the first line and the /* at the beginning of the second line are not necessary but are included for clarity. As the comment explains, the program reads standard input, converts TAB characters into the appropriate number of SPACEs, and writes the transformed input to standard output. Like many UNIX utilities, this program is a filter.

Figure 12-1. A simple C program: tabs.c (The line numbers are not part of the source code.)

[View full size image]

The comments at the top of tabs.c are followed by preprocessor directives, which are instructions for the C preprocessor. During the initial phase of compilation the C preprocessor expands these directives, making the program ready for the later stages of the compilation process. Preprocessor directives begin with the pound sign (#) and may optionally be preceded by SPACE and TAB characters.

Symbolic constants

You can use the #define preprocessor directive to define symbolic constants and macros. Symbolic constants are names that you can use in a program in place of constant values. For example, tabs.c uses a #define preprocessor directive to associate the symbolic constant TABSIZE with the constant 8. TABSIZE is then used in the program in place of the constant 8 as the distance between TAB stops. By convention the names of symbolic constants consist of all uppercase letters.

By defining symbolic names for constant values, you can make a program easier to read and easier to modify. If you later decide to change a constant, you need to change only the preprocessor directive; you do not need to change the value everywhere it occurs in the program. If you replace the #define directive for TABSIZE in Figure 12-1 with the following directive, the program will place TAB stops every four columns rather than every eight:

#define     TABSIZE    4

Macros

A symbolic constant, which is a type of macro, maps a symbolic name to replacement text. Macros are handy when the replacement text is needed at multiple points throughout the source code or when the definition of the macro is subject to change. The process of substituting the replacement text for the symbolic name is called macro expansion.

You can also use #define directives to define macros with arguments. Use of such a macro resembles a function call. Unlike C functions, however, macros are replaced with C code prior to compilation into object files.

The NEXTTAB macro computes the distance to the next TAB stop, given the current column position curcol:

#define NEXTTAB(curcol) (TABSIZE - ((curcol) % TABSIZE))

This definition uses the macro TABSIZE, whose definition must appear prior to NEXTTAB in the source code. The macro NEXTTAB could be used in tabs.c to assign a value to retval in the function findstop:

retval = NEXTTAB(*col);

Headers (include files)

When modules of a program use several macro definitions, the definitions are typically collected together in a single file called a header file or an include file. Although the C compiler does not place constraints on the names of header files, by convention they end in .h. The name of the header file is listed in an #include preprocessor directive in each program source file that uses any of the macros. The program in Figure 12-1 uses getchar and putchar, which are functions defined in stdio.h. The stdio.h header file defines a variety of general-purpose macros and is used by many C library functions.

The angle brackets (< and >) that surround stdio.h in tabs.c instruct the C preprocessor to look for the header file in a standard list of directories (such as /usr/include). To include a header file from another directory, enclose its pathname between double quotation marks. You can specify an absolute pathname within the double quotation marks or you can give a relative pathname. If you give a relative pathname, searching begins with the working directory and then moves to the same directories that are searched when the header file is surrounded by angle brackets.

You can also specify directories to be searched for header files by using the I option to the C compiler. Assume that you want to compile the program deriv.c, which contains the following preprocessor directive:

#include "eqns.h"

If the header file eqns.h is located in the subdirectory myincludes, you can compile deriv.c with the I option to tell the C preprocessor to look for eqns.h there:

$ gcc -I./myincludes deriv.c

When the C preprocessor encounters the #include directive in the deriv.c file, it will look for eqns.h in the myincludes subdirectory of the working directory.

Tip: Use relative pathnames for include files

Using absolute pathnames for include files does not work if the location of the header file within the filesystem changes. Using relative pathnames for header files works as long as the location of the header file relative to the working directory remains the same. Relative pathnames also work with the I option on the gcc command line and allow header files to be moved.

Function prototype

Preceding the definition of the function main is a function prototype. This declaration tells the compiler what type a function returns, how many arguments a function expects, and what the types of those arguments are. In tabs.c the prototype for the function findstop informs the compiler that findstop returns type int and that it expects a single argument of type pointer to int:

int findstop(int *);

Once the compiler has seen this declaration, it can detect and flag inconsistencies in the definition and the uses of the function. As an example, suppose that the reference to findstop in tabs.c was replaced with the following statement:

inc = findstop();

The prototype for findstop would cause the compiler to detect a missing argument and issue an error message. You could then easily fix the problem. When a function is present in a separate source file or is defined after it is referenced in a source file (as findstop is in the example), the function prototype helps the compiler check that the function is being called properly. Without the prototype, the compiler would not issue an error message and the problem might manifest itself as unexpected behavior during execution. At this late point, finding the bug might be difficult and time-consuming.

Functions

Although you can name most C functions anything you want, each program must have exactly one function named main. The function main is the control module: A program begins execution with the function main, which typically calls other functions, which in turn may call still other functions, and so forth. By putting different operations into separate functions, you can make a program easier to read and maintain. For example, the program in Figure 12-1 uses the function findstop to compute the distance to the next TAB stop. Although the few statements of findstop could easily have been included in the main function, isolating them in a separate function draws attention to a key computation.

Functions can make both development and maintenance of the program more efficient. By putting a frequently used code segment into a function, you avoid entering the same code into the program over and over again. When you later want to modify the code, you need change it only once.

If a program is long and includes several functions, you may want to split it into two or more files. Regardless of its size, you may want to place logically distinct parts of a program in separate files. A C program can be split into any number of different files; however, each function must be wholly contained within a single file.

Tip: Use a header file for multiple source files

When you are creating a program that takes advantage of multiple source files, put #define preprocessor directives into a header file. Then use an include statement with the name of the header file in any source file that uses the directives.

Compiling and Linking a C Program

To compile tabs.c and create an executable file named a.out, give the following command:

$ gcc tabs.c

The gcc utility calls the C preprocessor, the C compiler, the assembler, and the linker. Figure 12-2 shows these four components of the compilation process. The C preprocessor expands macro definitions and includes header files. The compilation phase creates assembly language code corresponding to the instructions in the source file. Then the assembler creates machine-readable object code. One object file is created for each source file. Each object file has the same name as the source file, except that the .c extension is replaced with .o. After successfully completing all phases of the compilation process for a program, the C compiler creates the executable file and then removes any temporary .o files.

Figure 12-2. The compilation process

During the final phase of the compilation process, the linker searches specified libraries for functions the program uses and combines object modules for those functions with the program's object modules. By default the C compiler links the standard C library libc.dylib (found in /usr/lib), which contains functions that handle input and output and provides many other general-purpose capabilities. If you want the linker to search other libraries, you must use the l (lowercase "l") option to specify the libraries on the command line. Unlike most options to Mac OS X system utilities, the l option does not come before all filenames on the command line but rather appears after the filenames of all modules that it applies to. In the next example, the C compiler searches the math library libm.dylib (also found in /usr/lib):

$ gcc calc.c -lm

The l option uses abbreviations for library names, appending the letter following l to lib and adding a .dylib or .a extension. The m in the example stands for libm.dylib.

Using the same naming mechanism, you can have a graphics library named libgraphics.a, which can be linked with the following command:

$ gcc pgm.c -lgraphics

When you use this convention to name libraries, gcc knows to search for them in /usr/lib. You can also have gcc search other directories by using the L option:

$ gcc pgm.c -L. -L/usr/X11R6/lib -lgraphics

The preceding command causes gcc to search for the library file libgraphics.a in the working directory and in /usr/X11R6/lib before searching /usr/lib.

As the last step of the compilation process, the linker creates an executable file named a.out unless you specify a different filename with the o option. Object files are deleted after the executable is created.

Mach-O format

You may occasionally encounter references to the a.out format, an old UNIX binary format. Mac OS X uses the Mach-O format for binaries, not the a.out format, in spite of the filename. Use the file utility (page 726) to display the format of the executable that gcc generates:

$ file a.out
a.out: Mach-O executable ppc

In the next example, the O3 option causes gcc to use the C compiler optimizer. The optimizer makes object code more efficient so that the executable program runs more quickly. Optimization has many facets, including locating frequently used variables and taking advantage of processor-specific features. The number after the O indicates the level of optimization, where a higher number specifies more optimization. See the gcc info page for specifics. The following example also shows that the .o files are not present after a.out is created:

$ ls
acctspay.c acctsrec.c ledger.c
$ gcc -O3 ledger.c acctspay.c acctsrec.c
$ ls
a.out acctspay.c acctsrec.c ledger.c

You can use the executable a.out in the same way you use shell scripts and other programs: by typing its name on the command line. The program in Figure 12-1 on page 481 expects to read from standard input, so once you have created the executable a.out, you can use a command such as the following to run it:

$ ./a.out < mymemo

If you want to save the a.out file, you should change its name to a more descriptive one. Otherwise, you might accidentally overwrite it during a later compilation.

$ mv a.out accounting

To save yourself the trouble of renaming an a.out file, you can specify the name of the executable file via the gcc command. The o option causes the C compiler to give the executable the name you specify rather than a.out. In the next example, the executable is named accounting:

$ gcc -o accounting ledger.c acctspay.c acctsrec.c

If accounting does not require arguments, you can run it with the following command:

$ accounting

You can suppress the linking phase of compilation by using the c option with the gcc command. The c option does not treat unresolved external references as errors; this capability enables you to compile and debug the syntax of the modules of a program as you create them. Once you have compiled and debugged all of the modules, you can run gcc again with the object files as arguments to produce an executable program. In the next example, gcc produces three object files but no executable:

$ gcc -c ledger.c acctspay.c acctsrec.c
$ ls
acctspay.c acctspay.o acctsrec.c acctsrec.o ledger.c ledger.o

If you then run gcc again and name the object files on the command line, gcc will produce the executable. Because it recognizes the filename extension .o, the C compiler knows that the files need only to be linked. You can also include both .c and .o files on a single command line:

$ gcc -o accounting ledger.o acctspay.c acctsrec.o

The C compiler recognizes that the .c file needs to be preprocessed and compiled, whereas the .o files do not. The C compiler also accepts assembly language files ending in .s and assembles and links them. This feature makes it easy to modify and recompile a program.

You can use separate files to divide a project into functional groups. For instance, you might put graphics routines in one file, string functions in another file, and database calls in a third file. Having multiple files can enable several engineers to work on the same project concurrently and can speed up compilation. For example, if all functions are in one file and you make a change to one of the functions, the compiler must recompile all of the functions in the file. Thus the entire program will be recompiled, which may take considerable time even if you made only a small change. When you use separate files, only the file that you change must be recompiled. For large programs with many source files (for example, the C compiler or emacs), the time lost by recompiling one huge file for every small change would be enormous. For more information refer to "make: Keeps a Set of Programs Current" on page 489.

Tip: What not to name a program

Do not name a program test or any other name of a builtin or other executable on the local system. If you do, you will likely execute the builtin or other program instead of the program you intend to run. Use which (page 58) to determine which program you will run when you give a command.


Using Shared Libraries

Most modern operating systems use shared libraries, also called dynamic libraries. These libraries are not linked into a program at compile time but rather are loaded when the program starts (or later in some cases). The names of files housing shared libraries have filename extensions of .dylib (dynamic library)for example, libc.dylib. Sometimes libaaa.dylib is a symbolic link to libaaa.dylib.x, where x is a small number representing the version of the library. Many of these libraries are kept in /usr/lib: A typical Mac OS X installation has more than 150 shared libraries in /usr/lib and more than 30 in /usr/X11R6/lib. Applications can have their own shared libraries. For example, the gcc compiler might keep its libraries in /usr/lib/gcc-lib/powerpc-apple-darwin8/4.0.0.

Archived libraries

In contrast to shared libraries are the older, statically linked libraries (with a .a filename extension), also called archived libraries. Archived libraries are added to the executable file during the last (linking) phase of compilation. Their addition can make a program run slightly faster the first time it is run, albeit at the expense of program maintainability and size. Taken together, the combined size of several executables that use a shared library and the size of the shared library are smaller than the combined size of the same executables with static libraries. When a running program has already loaded a dynamic library, a second program that requires the same dynamic library starts slightly faster.

Reducing memory usage and increasing maintainability are the primary reasons for using shared object libraries; thanks to these advantages, they have largely replaced statically linked libraries as the library type of choice. Consider what happens when you discover an error in a library. With a static library, you need to relink every program that uses the library once the library has been fixed and recompiled. With a dynamic library, you need to fix and recompile only the library itself.

Shared object libraries also make dynamic loading of program libraries on the fly possible (for example, perl, python, and tcl extensions and modules). The Apache (HTTP) Web server specifies modules in the httpd.conf file and loads them as needed.

otool

The otool (object tool) utility with the L option tells you which shared libraries a program needs. The following example shows that cp uses libSystem.B.dylib, one of the system libraries:

$ otool -L /bin/cp
/bin/cp:
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.0.0)

Refer to page 805 for more information on otool.

The compiler needs to see the shared libraries at link time to make sure that the needed functions and procedures are present as promised by the header (.h) files. Use the L option to tell the compile-time linker to look in the directory mylib for shared or static libraries: L mylib. Unlike the search path, L can use relative pathnames such as L ../liba handy feature when a program builds its own shared library. Each shared library contains a pathname telling the linker where to look for it; this path is recorded in the executable. When the program is run, dyld (the dynamic link editor) looks for each library in the recorded location. If dyld does not find a library in that location, dyld searches for it; see "Fixing Broken Binaries" on page 488. You can repeat the L option multiple times on the link line.

Frameworks

Mac OS X uses frameworks, which are similar to shared libraries. Frameworks differ from shared libraries in that they can contain resources, such as headers, in addition to code. Shared libraries are typically found in /usr/lib, while frameworks are usually found in /System/Library/Frameworks. Frameworks are specified with the framework option to the compiler; shared libraries are specified with l.

Fixing Broken Binaries

The command line search path is a fairly new idea. The search path was traditionally specified by the DYLD_LIBRARY_PATH environment variable. This variable has the same format as PATH (page 285). The directories in DYLD_LIBRARY_PATH are normally searched before the usual library locations. A number of other variables can affect the behavior of dyld. See the dyld man page for details.

The use of DYLD_LIBRARY_PATH brings up several problems. Because only one environment variable exists, it must be shared among all programs. If two programs have the same name for a library or use different, incompatible versions of the same library, only the first will be found. As a result one of the programs will not run oreven worsewill not run correctly.

Security: DYLD_LIBRARY_PATH

Under certain circumstances a malicious user can create a Trojan horse named libc.dylib and place it in a directory that is searched before /usr/lib (any directory that appears in DYLD_LIBRARY_PATH before /usr/lib). The fake libc will then be used instead of the real libc.

Wrappers

DYLD_LIBRARY_PATH still has its place in the scripts, called wrappers, that are used to fix broken binaries. Suppose that the broken binary bb uses the shared library libbb.so, which you want to put in /opt/bb/lib and not in /usr/lib, as the bb programmer requested. The command otool L bb will tell you which libraries are missing. Not a problem: Rename bb to bb.broken, and create a /bin/sh wrapper named bb.

$ cat bb
#!/bin/sh
DYLD_LIBRARY_PATH=/opt/bb/lib
export DYLD_LIBRARY_PATH
exec bb.broken "$@"

(Using $@ rather than $* preserves SPACEs in the parameters; see page 566.) A wrapper can also allow you to install programs in arbitrary locations.

If dyld does not find a library anywhere in DYLD_LIBRARY_PATH, dyld searches DYLD_FALLBACK_LIBRARY_PATH ($HOME/lib:/usr/local/lib:/lib:/usr/lib is the default). Frameworks are searched for in DYLD_FRAMEWORK_PATH and DYLD_FALLBACK_FRAMEWORK_PATH.

Creating Shared Libraries

Building a dynamically loadable shared library is not a trivial matter: It involves setting up reentrant function calls, defining a library entrance routine, and performing other tasks. When you want to create a shared object library, you must compile the source files with the dynamiclib option to gcc. Shared libraries can depend on other shared libraries. The best resource for investigating shared library construction and usage is existing code on the Internet. For example, you can look at the source files for zlib at www.gzip.org/zlib.

C++

C++ files have special needs, and libraries (shared or not) often have to be made by the compiler rather than ld or ar.


make: Keeps a Set of Programs Current

Tip: This section covers the GNU make program

This section describes the GNU make program that is distributed with Mac OS X. Other make tools (BSN make, GNUstep make, Borland make, and so on) are available as well as similar tools such as ant (the Apache build tool) and Xcode. Makefiles created for GNU make are often incompatible with other make tools, which can be problematic if you are trying to compile code targeted for another platform.

In a large program with many source and header files, the files typically depend on one another in complex ways. When you change a file that other files depend on, you must recompile all dependent files. For example, you might have several source files, all of which use a single header file. When you change the header file, you must recompile each of the source files. The header file might depend on other header files, and so forth. Figure 12-3 shows a simple example of dependency relationships. Each arrow in this figure points from a file to another file that depends on it.

Figure 12-3. Dependency graph for the target form

When you are working on a large program, it can be difficult, time-consuming, and tedious to determine which modules need to be recompiled because of their dependency relationships. The make utility automates this process.

Dependency lines: target files and prerequisite files

At its simplest, make looks at dependency lines in a file named makefile or Makefile in the working directory. The dependency lines indicate relationships among files, specifying a target file that depends on one or more prerequisite files. If you have modified any of the prerequisite files more recently than their target file, make updates the target file based on construction commands that follow the dependency line. The make utility normally stops if it encounters an error during the construction process.

The file containing the updating information for the make utility is called a makefile. (See page 479 for a trivial example.) A simple makefile has the following syntax:

target: prerequisite-list
TAB construction-commands

The dependency line consists of the target and the prerequisite-list, separated by a colon. Each construction-commands line (you may have more than one) must start with a TAB and must follow the dependency line. Long lines can be continued with a BACKSLASH (\) as the last character on the line.

The target is the name of the file that depends on the files in the prerequisite-list. The construction-commands are regular shell commands that construct (usually compile and/or link) the target file. The make utility executes the construction-commands when the modification time of one or more files in the prerequisite-list is more recent than that of the target file.

The following example shows the dependency line and construction commands for the file named form in Figure 12-3. The form file depends on the prerequisites size.o and length.o. An appropriate gcc command constructs the target:

form: size.o length.o
TAB     gcc -o form size.o length.o

Each of the prerequisites on one dependency line can be a target on another dependency line. For example, both size.o and length.o are targets on other dependency lines. Although the example in Figure 12-3 is simple, the nesting of dependency specifications can create a complex hierarchy that dictates relationships among many files.

The following makefile (named Makefile) corresponds to the complete dependency structure shown in Figure 12-3. The executable file form depends on two object files, and each of the object files depends on its respective source file and a header file, form.h. In turn, form.h depends on two other header files.

$ cat Makefile
form: size.o length.o
       gcc -o form size.o length.o
size.o: size.c form.h
       gcc -c size.c
length.o: length.c form.h
       gcc -c length.c
form.h: num.h table.h
       cat num.h table.h > form.h

Although the last line would not normally be seen in a makefile, it illustrates the point that you can put any shell command on a construction line. Because the shell processes makefiles, the command line should be one that you could enter in response to a shell prompt.

The following command builds the default target form if any of its prerequisites are more recent than their corresponding targets or if any of the targets do not exist:

$ make

Thus, if the file form has been deleted, make will rebuild it, regardless of the modification dates of its prerequisite files. The first target in a makefile is the default and is built when you call make without any arguments.

If you want make to rebuild a target other than the first in the makefile, you must provide that target as an argument to make. The following command rebuilds only form.h if it does not exist or if its prerequisites were more recently modified than the target:

$ make form.h

Implied Dependencies

You can rely on implied dependencies and construction commands to facilitate the job of writing a makefile. For instance, if you do not include a dependency line for an object file, make assumes that it depends on a compiler or assembler source code file. Thus, if a prerequisite for a target file is xxx.o and no dependency line identifies xxx.o as a target, make looks at the filename extension to determine how to build the .o file. If it finds an appropriate source file, make provides a default construction command line that calls the proper compiler or the assembler to create the object file. Table 12-1 lists some filename extensions that make recognizes and the type of file that corresponds to each filename extension.

Table 12-1. Filename extensions

Filename with extension

Type of file

filename.c

C programming language source code

filename.C, filename.cc, filename.cxx, filename.c++, filename.cpp

C++ programming language source code

filename.h

Header file

filename.l

flex, lex lexical analyzer generator source code

filename.m

Objective-C programming language source code

filename.o

Object module

filename.s

Assembler code

filename.sh

Shell script

filename.y

bison, yacc parser generator source code

C and C++ are traditional programming languages that are available with Mac OS X. The bison and flex tools create command languages.

In the next example, a makefile keeps the file named compute up-to-date. The make utility ignores any line that begins with a pound sign (#). Thus the first three lines of the following makefile are comment lines. The first dependency line shows that compute depends on two object files: compute.o and calc.o. The corresponding construction line gives the command that make needs to produce compute. The second dependency line shows that compute.o depends not only on its C source file but also on the compute.h header file. The construction line for compute.o uses the C compiler optimizer (O3 option). The third set of dependency and construction lines is not required. In their absence, make infers that calc.o depends on calc.c and produces the command line needed for the compilation.

$ cat Makefile
# Makefile for compute
compute: compute.o calc.o
       gcc -o compute compute.o calc.o

compute.o: compute.c compute.h
       gcc -c -O3 compute.c

calc.o: calc.c
       gcc -c calc.c

clean:
       rm *.o *~

There are no prerequisites for clean, the last target. This target is commonly used to get rid of extraneous files that may be out-of-date or no longer needed, such as .o files.

Below are some sample executions of make based on the previous makefile. As the ls command shows, compute.o, calc.o, and compute are not up-to-date. Consequently the make command runs the construction commands that re-create them.

$ ls -ltr
total 22
-rw-rw---- 1 alex alex  311 Jun 21 15:56 makefile
-rw-rw---- 1 alex alex  354 Jun 21 16:02 calc.o
-rwxrwx--- 1 alex alex 6337 Jun 21 16:04 compute
-rw-rw---- 1 alex alex   49 Jun 21 16:04 compute.h
-rw-rw---- 1 alex alex  880 Jun 21 16:04 compute.o
-rw-rw---- 1 alex alex  780 Jun 21 18:20 compute.c
-rw-rw---- 1 alex alex  179 Jun 21 18:20 calc.c
$ make
gcc -c -O3 compute.c
gcc -c calc.c
gcc -o compute compute.o calc.o

If you run make once and then run it again without making any changes to the prerequisite files, make indicates that the program is up-to-date and does not execute any commands:

$ make
make: 'compute' is up to date.

touch

The next example uses touch (page 877) to change the modification time of a prerequisite file. This simulation shows what happens when you alter the file. The make utility executes only the commands necessary to bring the out-of-date targets up-to-date:

$ touch calc.c
$ make
gcc -c calc.c
gcc -o compute compute.o calc.o

In the next example, touch changes the modification time of compute.h. The make utility re-creates compute.o because it depends on compute.h and re-creates the executable because it depends on compute.o:

$ touch compute.h
$ make
gcc -c -O3 compute.c
gcc -o compute compute.o calc.o

n

If you want to see what make would do if you ran it, run make with the n (no execute) option. This option displays the commands that make would execute but it does not execute them.

t

As these examples illustrate, touch is useful when you want to fool make either into recompiling programs or into not recompiling them. You can use touch to update the modification times of all source files so that make considers nothing to be up-to-date; make will then recompile everything. Alternatively, you can use touch or the t option to make to touch all relevant files; make then considers everything to be up-to-date. Using touch in this manner is useful if the modification times of files have changed yet the files remain up-to-date (as can happen when you copy a set of files from one directory to another).

The following example uses make n several times to see what make would do if you gave a make command. The first command shows that the target, compute, is up-to-date. Next touch makes the modification dates on all *.c files more recent than their targets, and make n shows what make would do if you called it without the n option. The make t command then brings all the targets up-to-date. The final make n confirms that compute is up-to-date.

$ make -n
make: 'compute' is up to date.
$ touch *.c
$ make -n
gcc -c -O3 compute.c
gcc -c calc.c
gcc -o compute compute.o calc.o
$ make -t
touch compute.o
touch calc.o
touch compute
$ make -n
make: 'compute' is up to date.

j

The j (jobs) option performs a number of tasks in parallel; the numeric argument to j specifies the number of jobs or processes. Most make tasks hit the disk first and then the CPU, resulting in CPU usage dropping between compilations. On a multiprocessor system, you can reduce CPU usage by using make j n, where n is the number of CPUs plus 1. Running tasks in parallel can significantly reduce the build time for a large project.

Once you are satisfied with the program you have created, you can use the makefile to remove extraneous files. It is helpful to keep intermediate files around while you are writing and debugging a program so that you need to rebuild only the ones that change. When you will not be working on the program for a while, you can release the disk space occupied by the extra files. Using a clean target in a makefile means that you do not have to remember all the little pieces that can safely be deleted. The next example simply removes all object (.o) files and all files with filenames that end with a tilde (~):

$ make clean
rm *.o *~

See page 786 for an example of a more complex makefile.

Optional: Macros

The make utility's macro facility enables you to create and use macros within a makefile. The syntax of a macro definition is

ID = list

Replace ID with an identifying name, and replace list with a list of filenames. After this macro definition, $(ID) represents list in the makefile.

With a macro you can compile a program with any of several C compilers by making only a minor change to the makefile. By using the CC macro and replacing all occurrences of gcc in the makefile on page 492 with $(CC), for example, you need to assign a value only to CC to use the compiler of your choice:

$ cat Makefile
#Makefile for compute
CC=gcc
compute: compute.o calc.o
       $(CC) -o compute compute.o calc.o

compute.o: compute.c compute.h
       $(CC) -c -O3 compute.c

calc.o: calc.c
       $(CC) -c calc.c

clean:
       rm *.o

This example assumes that the compiler/loader flags are the same across compilers/loaders. In a more complex situation, you need to create macros for these flags or use the default values. Commercial compilers are also available for Mac OS X. They generally have their own graphical environments comparable to Xcode. If you do not assign a value to the CC macro, it defaults to gcc under Mac OS X. The CC macro invokes the C compiler with only the options that you specify.

Several other macro definitions are commonly used. The CFLAGS macro sends arguments to the C compiler, LDFLAGS sends arguments to the linker (ld, or gcc o), and CPPFLAGS sends arguments to the C preprocessor and programs that use it, including gcc. The COMPILE.c macro expands to $(CC) c $(CFLAGS) $(CPPFLAGS). The LINK.c macro expands to $(CC) $(CFLAGS) $(CPPFLAGS) $(LDFLAGS).

By default make invokes the C compiler without any options (except the c option when it is appropriate to compile but not link a file). You can use the CFLAGS macro definition to cause make to call the C compiler with specific options. Replace options with the options you want to use:

CFLAGS = options

The following makefile uses macros as well as implied dependencies and constructions:

# makefile: report, print, printf, printh
#
CC=gcc
CFLAGS = -O3
FILES = in.c out.c ratio.c process.c tally.c
OBJECTS = in.o out.o ratio.o process.o tally.o
HEADERS = names.h companies.h conventions.h

report: $(OBJECTS)
       $(LINK.c) -o report $(OBJECTS)

ratio.o: $(HEADERS)
process.o: $(HEADERS)

tally.o: $(HEADERS)

print:
    pr $(FILES) $(HEADERS) | lpr

printf:
    pr $(FILES) | lpr

printh:
    pr $(HEADERS) | lpr

Following the comment lines in this example, the makefile uses the CFLAGS macro to cause make always to use the optimizer (O3 option) when it invokes the C compiler as the result of an implied construction. A construction line in a makefile overrides the corresponding implied construction line, if one exists. If you want to apply a macro to a construction command, you must include the macro in that command; see OBJECTS in the construction command for the report target. Following CFLAGS, the makefile defines the FILES, OBJECTS, and HEADERS macros. Each of these macros defines a list of files.

The first dependency line in the preceding example shows that report depends on the list of files that OBJECTS defines. The corresponding construction line uses the LINK.c macro to link the OBJECTS and create an executable file named report. If you specify any LDFLAGS, they are used in this step.

The next three dependency lines show that three object files depend on the list of files that HEADERS defines. Because there are no construction lines, make looks for a source code file corresponding to each object file and compiles it. These three dependency lines ensure that the object files are recompiled if any header files change.

You can combine several targets on one dependency line, so these three dependency lines could have been combined into one line:

ratio.o process.o tally.o: $(HEADERS)

The three final dependency lines in the preceding example send source and header files to the printer. They have nothing to do with compiling the report file. None of these targets (print, printf, and printh) depends on anything. When you call one of these targets from the command line, make executes the construction line following it. The following command prints all the source files that FILES defines:

$ make printf

You can override macros in a makefile by specifying them on the command line. The following command adds debugging symbols to all object files:

$ make CFLAGS=-g ...


Debugging C Programs

The C compiler is liberal about the kinds of constructs it allows in programs. In keeping with the UNIX philosophy that "no news is good news" and that the user knows what is best, gcc, like many other OS X utilities, accepts almost any construct that is logically possible according to the definition of the language. Although this approach gives the programmer a great deal of flexibility and control, it can make debugging difficult.

Figure 12-4 shows badtabs.c, a flawed version of the tabs.c program discussed earlier. It contains some errors and does not run properly. This section uses this program to illustrate some debugging techniques.

Figure 12-4. The badtabs.c program (The line numbers are not part of the source code; the arrows point to errors in the program.)

[View full size image]

In the following example, badtabs.c is compiled and then run with input from the testtabs file. Inspection of the output shows that the TAB character has not been replaced with the proper number of SPACEs:

$ gcc -o badtabs badtabs.c
$ cat testtabs
abcTABxyz
$ badtabs < testtabs
abc   xyz

One way to debug a C program is to insert print statements at critical points throughout the source code. To learn more about the behavior of badtabs.c when it runs, you can replace the contents of the switch statement with

case '\t':              /* c is a tab */
    fprintf(stderr, "before call to findstop, posn is %d\n", posn);
    inc = findstop(&posn);
    fprintf(stderr, "after call to findstop, posn is %d\n", posn);
    for( ; inc > 0; inc-- )
        putchar(' ');
    break;
case '\n':              /* c is a newline */
    fprintf(stderr, "got a newline\n");
    putchar(c);
    posn = 0;
    break;
default:                /* c is anything else */
    fprintf(stderr, "got another character\n");
    putchar(c);
    posn++;
    break;

The fprintf statements in this code send their messages to standard error. Thus, if you redirect standard output of this program, it will not be interspersed with the output sent to standard error. The next example demonstrates the operation of this program on the input file testtabs:

$ gcc -o badtabs badtabs.c
$ badtabs < testtabs > testspaces
got another character
got another character
got another character
before call to findstop, posn is 3
after call to findstop, posn is 3
got another character
got another character
got another character
got a newline
$ cat testspaces
abcTABxyz

The fprintf statements provide additional information about the execution of tabs.c. The value of the variable posn is not incremented in findstop, as it should be. This clue might be enough to lead you to the bug in the program. If not, you might attempt to "corner" the offending code by inserting print statements in findstop.

For simple programs or when you have an idea of what is wrong with a program, adding print statements that trace the execution of the code can often help you discover the problem quickly. A better strategy may be to take advantage of the tools that Mac OS X provides to help you debug programs.

gcc: Compiler Warning Options

The gcc compiler includes many of the features of lint, the classic C program verifier, and then some. (The lint utility is not available under OS X; use splint [secure programming lint; www.splint.org] instead.) The gcc compiler can identify many C program constructs that pose potential problems, even for programs that conform to the syntax rules of the language. For instance, you can request that the compiler report a variable that is declared but not used, a comment that is not properly terminated, or a function that returns a type not permitted in older versions of C. Options that enable this stricter compiler behavior all begin with the uppercase letter W (Warning).

Among the W options is a class of warnings that typically result from programmer carelessness or inexperience (see Table 12-2). The constructs that generate these warnings are generally easy to fix and easy to avoid.

Table 12-2. gcc W options

Option

Reports an error when

Wimplicit

A function or parameter is not explicitly declared

Wreturn-type

A function that is not void does not return a value or the type of a function defaults to int

Wunused

A variable is declared but not used

Wcomment

The characters /*, which normally begin a comment, occur within a comment

Wformat

Certain input/output statements contain format specifications that do not match the arguments

The Wall option displays warnings about all the errors listed in Table 12-2, along with other, similar errors.

The program badtabs.c is syntactically correct: It compiles without generating an error. However, if you compile it (c causes gcc to compile but not to link) with the Wall option, gcc identifies several problems. (Warning messages do not stop the program from compiling, whereas error messages do.)

$ gcc -c -Wall badtabs.c
badtabs.c:47: warning: '/*' within comment
badtabs.c:11: warning: return-type defaults to 'int'
badtabs.c: In function 'main':
badtabs.c:34: warning: control reaches end of non-void function
badtabs.c: In function 'findstop':
badtabs.c:40: warning: unused variable 'colindex'
badtabs.c:49: warning: control reaches end of non-void function

The first warning message references line 47. Inspection of the code for badtabs.c around that line reveals a comment that is not properly terminated. The compiler sees the string /* in the following line as the beginning of a comment:

/* increment argument (current column position) to next tabstop */

However, because the characters * and / at the end of the line are separated by a SPACE, they do not signify the end of the comment to the compiler. Instead the compiler interprets all of the statementsincluding the statement that increments the argumentthrough the string * / at the very end of the findstop function as part of the comment. After you remove the SPACE between the characters * and /, badtabs produces the correct output.

The next few paragraphs discuss the remaining warning messages. Although most do not cause problems in the execution of badtabs, you can generally improve a program by rewriting those parts of the code that produce such warnings.

Because the definition of the function main does not include an explicit type, the compiler assumes type int, the default. This results in the warning message referencing line 11 in badtabs.c, the top of the function main. An additional warning is given when the compiler encounters the end of the function main (line 34) without seeing a value returned.

If a program runs successfully, by convention it should return a zero value; if no value is returned, the exit code is undefined. Although many C programs do not return a value, this oversight can cause problems when the program is executed. When you add the following statement at the end of the function main in badtabs.c, the warning referencing line 34 disappears:

return 0;

Line 40 of badtabs.c contains the definition for the local variable colindex in the function findstop. A warning message references this line because the colindex variable is never used. Removing its declaration eliminates the warning message.

The final warning message, referencing line 49, results from the improperly terminated comment discussed earlier. The compiler issues the warning message because it never sees a return statement in findstop. (The compiler ignores commented text.) Because the function findstop returns type int, the compiler expects to find a return statement before reaching the end of the function. The warning disappears when the comment is properly terminated.

Many other W options are available with the gcc compiler. The ones not covered in the Wall class often deal with portability differences; modifying the code causing these warnings may not be appropriate. The warnings usually result from programs that are written in different C dialects as well as from constructs that may not work well with other (especially older) C compilers. The pedantic-errors option turns warnings into errors, causing a build to fail if it contains items that would generate warnings. To learn more about these and other warning options, refer to the gcc info page.

Symbolic Debuggers

Many debuggers are available to tackle problems that evade simpler debugging methods such as print statements and compiler warning options. These debuggers include gdb, as well as a number of graphical debuggers, that are available from the Web (refer to Appendix B). Such high-level symbolic debuggers enable you to analyze the execution of a program in terms of C language statements. They also provide a lower-level view for analyzing the execution of a program in terms of the machine instructions. Apple's Xcode provides a graphical front end to gdb.

A debugger enables you to monitor and control the execution of a program. You can step through a program line by line while you examine the state of the execution environment.

Core dumps

A debugger also allows you to examine core files. Core files are stored in the /cores directory and named core.pid. When a serious error occurs during the execution of a program, the operating system can create a core file containing information about the state of the program and the system when the error occurred. This file comprises a dump of the computer's memory (it was previously called core memoryhence the term core dump) that was being used by the program. To conserve disk space, OS X does not save core files automatically. You can change this behavior by editing /etc/hostconfig. Add the line COREDUMPS=YES, remove other lines containing COREDUMPS, and reboot. You can use the ulimit builtin to allow core files to be saved for a single shell session. If you are running bash, the following command allows core files of unlimited size to be saved to disk:

$ ulimit -c unlimited

The operating system advises you when it dumps core. You can use a symbolic debugger to read information from the core file to identify the line in the program where the error occurred, to check the values of variables at that point, and so forth. Because core files tend to be large and take up disk space, be sure to remove these files when you no longer need them.

gdb: Symbolic Debugger

This section explains how to use the GNU gdb debugger from the command line. Because gdb is the back end of Xcode's graphical debugger, you can also use it directly from Xcode. Other symbolic debuggers offer a different interface but operate in a similar manner. To take full advantage of a symbolic debugger with a program, you must compile the program with the g option, which causes gcc to generate additional information that the debugger uses. This information includes a symbol tablea list of variable names used in the program and their associated values. Without the symbol table information, the debugger cannot display the values and types of variables. If a program is compiled without the g option, gdb cannot identify source code lines by number, as many gdb commands require.

Tip: Always use g

It is a good idea to use the g option even when you are releasing software. Including debugging symbols makes a binary a bit bigger. Debugging symbols do not make a program run more slowly, but they do make it much easier to find problems identified by users.

Tip: Avoid using optimization flags with the debugger

Limit the optimization flags to O or O2 when you compile a program for debugging. Because debugging and optimizing inherently have different goals, it may be best to avoid combining the two operations.

Tip: Optimization should work

Turning optimization off completely can sometimes eliminate errors. Eliminating errors in this way should not be seen as a permanent solution, however. When optimization is not enabled, the compiler may automatically initialize variables and perform certain other checks for you, resulting in more stable code. Correct code should work correctly when compiled with at least O and almost certainly with O2. The O3 setting often includes experimental optimizations; it may not generate correct code in all cases.

The following example uses the g option when creating the executable file tabs from the C program tabs.c, discussed at the beginning of this chapter:

$ gcc -g tabs.c -o tabs

Input for tabs is contained in the file testtabs, which consists of a single line:

$ cat testtabs
xyzTABabc

You cannot specify the input file to tabs when you call the debugger. Instead, you call the debugger and then specify the input file when you start execution with the run command.

To run the debugger on the sample executable, give the name of the executable file on the command line when you run gdb. You will see some introductory statements about gdb, followed by the gdb prompt [(gdb)]. At this point the debugger is ready to accept commands. The list command displays the first ten lines of source code. A subsequent list command displays the next ten lines of source code.

$ gdb tabs
GNU gdb 6.1-20040303 (Apple version gdb-384) (Mon Mar 21 00:05:26 GMT 2005)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "powerpc-apple-darwin".
(gdb) list
4       #include        <stdio.h>
5       #define         TABSIZE         8
6
7       /* prototype for function findstop */
8       int findstop(int *);
9
10      int main()
11      {
12      int c;          /* character read from stdin */
13      int posn = 0;   /* column position of character */
(gdb) list
14      int inc;        /* column increment to tab stop */
15
16      while ((c = getchar()) != EOF)
17              switch(c)
18                      {
19                      case '\t':              /*  c is a tab * /
20                              inc = findstop(&posn);
21                              for( ; inc > 0; inc-- )
22                                      putchar(' ');
23                              break;
(gdb)

One of the most important features of a debugger is its ability to run a program in a controlled environment. You can stop the program from running whenever you want. While it is stopped, you can check the state of an argument or variable. For example, you can give the break command a source code line number, an actual memory address, or a function name as an argument. The following command tells gdb to stop the process whenever the function findstop is called:

(gdb) break findstop
Breakpoint 1 at 0x2cac: file tabs.c, line 41.
(gdb)

The debugger acknowledges the request by displaying the breakpoint number, the hexadecimal memory address of the breakpoint, and the corresponding source code line number (41). The debugger numbers breakpoints in ascending order as you create them, starting with 1.

After setting a breakpoint you can issue a run command to start execution of tabs under the control of the debugger. The run command syntax allows you to use angle brackets to redirect input and output (just as the shells do). In the following example, the testtabs file is specified as input. When the process stops (at the breakpoint), you can use the print command to check the value of *col. The backtrace (or bt) command displays the function stack. In this example, the currently active function has been assigned the number 0. The function that called findstop (main) has been assigned the number 1.

(gdb) run < testtabs
Starting program: /Users/alex/book/12/tabs < testtabs

Breakpoint 1, findstop (col=0xbffffa00) at tabs.c:41
41      retval = (TABSIZE - (*col % TABSIZE));
(gdb) print *col
$1 = 3
(gdb) backtrace
#0 findstop (col=0xbffffa00) at tabs.c:41
#1 0x00002c0c in main () at tabs.c:20
(gdb)

You can examine anything in the current scopevariables and arguments in the active function as well as global variables. In the next example, the request to examine the value of the variable posn at breakpoint 1 results in an error. The error is generated because the variable posn is defined locally in the function main, not in the function findstop.

(gdb) print posn
No symbol "posn" in current context.

The up command changes the active function to the caller of the currently active function. Because main calls the function findstop, the function main becomes the active function when the up command is given. (The down command does the inverse.) The up command may be given an integer argument specifying the number of levels in the function stack to backtrack, with up 1 having the same meaning as up. (You can use the backtrace command to determine which argument to use with up.)

(gdb) up
#1 0x00002c0c in main () at tabs.c:20
20                              inc = findstop(&posn);
(gdb) print posn
$2 = 3
(gdb) print *col
No symbol "col" in current context.
(gdb)

The cont (continue) command causes the process to continue running from where it left off. The testtabs file contains only one line; thus the process finishes executing and the results appear on the screen. The debugger reports the exit code of the program. A cont command given after a program has finished executing reminds you that execution of the program is complete. The debugging session is then ended with a quit command.

(gdb) cont
Continuing.
abc     xyz

Program exited normally.
(gdb) cont
The program is not being run.
(gdb) quit
$

The gdb debugger supports many commands that are designed to make debugging easier. Type help at the (gdb) prompt to get a list of the command classes available under gdb:

(gdb) help
List of classes of commands:

aliases -- Aliases of other commands
breakpoints -- Making program stop at certain points
data-- Examining data
files -- Specifying and examining files
internals -- Maintenance commands
obscure -- Obscure features
running -- Running the program
stack -- Examining the stack
status -- Status inquiries
support -- Support facilities
tracepoints -- Tracing of program execution without stopping the program
user-defined -- User-defined commands

Type "help" followed by a class name for a list of commands in that class.
Type "help" followed by command name for full documentation.
Command name abbreviations are allowed if unambiguous.
(gdb)

As explained in the instructions following the list, entering help followed by the name of a command class or command name will display more information. The following listing shows the commands in the class data:

(gdb) help data
Examining data.

List of commands:

append -- Append target code/data to a local file
call -- Call a function in the program
delete display -- Cancel some expressions to be displayed when program stops
delete mem -- Delete memory region
disable display -- Disable some expressions to be displayed when program stops
disable mem -- Disable memory region
disassemble -- Disassemble a specified section of memory
display -- Print value of expression EXP each time the program stops
dump -- Dump target code/data to a local file
enable display -- Enable some expressions to be displayed when program stops
enable mem -- Enable memory region
inspect -- Same as "print" command
mem -- Define attributes for memory region
output -- Like "print" but don't put in value history and don't print newline
print -- Print value of expression EXP
print-object -- Ask an Objective-C object to print itself
printf -- Printf "printf format string"
ptype -- Print definition of type TYPE
restore -- Restore the contents of FILE to target memory
set -- Evaluate expression EXP and assign result to variable VAR
set variable -- Evaluate expression EXP and assign result to variable VAR
undisplay -- Cancel some expressions to be displayed when program stops
whatis -- Print data type of expression EXP
x -- Examine memory: x/FMT ADDRESS

Type "help" followed by command name for full documentation.
Command name abbreviations are allowed if unambiguous.
(gdb)

The following command requests information on the command whatis, which takes a variable name or other expression as an argument:

(gdb) help whatis
Print data type of expression EXP.

Graphical Symbolic Debuggers

Several graphical interfaces to gdb exist. The most widely used under Mac OS X is the one built into Xcode. Others include the GNU ddd debugger program (www.gnu.org/software/ddd).


Threads

A thread is a single sequential flow of control within a process. Threads are the basis for multithreaded programs, which allow a single program to control concurrently running threads, each performing a different task. Multithreaded programs generally use reentrant code (code that multiple threads can use simultaneously) and are most valuable when run on multiple-CPU machines. Under Mac OS X, multithreaded servers can provide a cleaner interface and may be easier to write than multiple server processes. When applied judiciously, multithreading can also serve as a lower-overhead replacement for the traditional fork-exec idiom for spawning processes. Documentation for threads includes the pthread man page (for the C interface) and the NSThread class documentation (for Objective-C). For more general information about threads, see the FAQ at tldp.org/FAQ/Threads-FAQ.

Tip: Multiple threads are not always better

If you write a multithreaded program with no clear goal or division of effort for a single-CPU system (for example, a parallel-server process), the resulting program will likely run more slowly than a nonthreaded program on the same system.


System Calls

Three fundamental responsibilities of the Mac OS X kernel are to control processes, manage the filesystem, and operate peripheral devices. As a programmer you have access to these kernel operations through system calls and library functions. This section discusses system calls at a general level; a detailed treatment is beyond the scope of this book.

As the name implies, a system call instructs the system (kernel) to perform some work directly on your behalf. The request tells the kernel what work needs to be done and includes the necessary arguments. For example, a system call to open a file includes the name of the file. A library routine is indirect; it issues system calls for you. The advantages of a library routine are that it may insulate you from the low-level details of kernel operations and that it has been written carefully to make sure that it performs efficiently.

For example, it is straightforward to use the standard I/O library function fprintf() to send text to standard output or standard error. Without this function, you would need to issue several system calls to achieve the same result. The calls to the library routines putchar() and getchar() in Figure 12-1 on page 481 ultimately use the write() and read() system calls to perform the I/O operations.

kTRace: traces System Calls

The ktrace utility is a debugging tool that displays a trace of all UNIX system calls made by a process or program. Because you do not need to recompile the program that you want to trace, you can use kTRace on binaries that you do not have source for.

System calls are events that take place at the interface (boundary) between user code and kernel code. Examining this boundary can help you isolate bugs, track down race conditions, and perform sanity checking. The ktrace utility records only UNIX system calls, not direct Mach system calls. The kTRace utility creates a log file named ktrace.out. Because the log file is not in human-readable format, you must use kdump to look at the system calls recorded by ktrace.

You can use the p pid option to kTRace to trace a running process. Include the i option to cause kTRace to trace child processes if the process you are tracing calls other programs. You must have root privileges to use ktrace to trace a setuid program.

Controlling Processes

When you enter a command at a shell prompt, the shell process calls the fork() system call to create a copy of itself (spawn a child) and then uses an exec() system call to overlay that copy in memory with a different program (the command you asked it to run). Table 12-3 lists system calls that affect processes.

Table 12-3. System calls: process control

System call

Function

fork()

Creates a copy of a process

exec()

Overlays a program in memory with another program

getpid()

Returns the PID number of the calling process

getppid()

Returns the PID number of the calling process's parent process

wait()

Causes the parent process to wait for the child process to finish running before it resumes execution

exit()

Causes a process to exit

nice()

Changes the priority of a process

kill()

Sends a signal to a process

Accessing the Filesystem

Many operations take place when a program reads from or writes to a file. The program needs to know where the file is located; so the filename must be converted to an inode number on the correct filesystem. Your access permissions must be checked not only for the file itself but also for all intervening directories in the path to the file. Because the file is not stored in one continuous piece on the disk, all disk blocks that contain pieces of the file must be located. The appropriate kernel device driver must be called to control the operation of the disk. Once the file has been found, the program may need to find a particular location within the file rather than working with it sequentially from beginning to end. Table 12-4 lists some of the most common system calls for filesystem operations.

Table 12-4. System calls: filesystem

System call

Function

stat()

Gets status information from an inode, such as the inode number, the device on which it is located, owner and group information, and the size of the file

lseek()

Moves to a position in the file

creat()

Creates a new file

open()

Opens an existing file

read()

Reads a file

write()

Writes a file

close()

Closes a file

unlink()

Unlinks a file (deletes a name reference to the inode)

chmod()

Changes file access permissions

chown()

Changes file ownership

Access to peripheral devices on a Mac OS X system is often handled through the filesystem interface. Peripheral devices are represented by one or more special files, usually located under /dev. When you read from or write to one of these special files, the kernel passes your request to the appropriate kernel device driver. As a result you can use the standard system calls and library routines to interact with these devices; you do not need to learn a new set of specialized functions. This ability is one of the most powerful features of a UNIX system because it allows users to use the same basic utilities on a wide range of devices. Under OS X, only some devices (those with BSD kernel interfaces) are accessible through files in /dev. Devices that use IOKit drivers are accessible through the Platform Expert kernel component.

The availability of standard system calls and library routines is the key to the portability of UNIX tools. For example, as an applications programmer, you can rely on the read() and write() system calls working the same way on different versions of UNIX and UNIX-like systems, and on different types of computers. The systems programmer who writes a device driver or ports the kernel to run on a new computer, however, must understand the details at their lowest level.


Source Code Management

When you work on a project involving many files that evolve over long periods of time, it can be difficult to keep track of the different versions of the files, particularly if several people are updating the files. This problem crops up regularly in large software development projects. Source code and documentation files change frequently as you fix bugs, enhance programs, and release new versions of the software. The task becomes even more complex when more than one version of each file is active. In many situations customers are using one version of a file while a newer version is being modified. You can easily lose track of the versions and accidentally undo changes or duplicate earlier work.

To help avoid these kinds of problems, Mac OS X includes CVS (Concurrent Versions System; see developer.apple.com/internet/opensource/cvsoverview.html and www.nongnu.org/cvs) for managing and tracking changes to files. Although CVS can be used on any file, it is most often employed to manage source code and software documentation. CVS is based on RCS (GNU's Revision Control System) and is designed to control the concurrent access and modification of source files by multiple users.

An OS Xspecific graphical front end to CVS named CVL (Concurrent Versions Librarian) simplifies the use of CVS, especially if you do not use it frequently enough to memorize its many commands and options. CVL is not provided with OS X; you can download it from www.sente.ch/software/cvl. TkCVS (page 518) is another GUI front end to CVS. It is available for Mac OS X; you can download it from www.twobarleycorns.net/tkcvs.html.

CVS controls who is allowed to update files. For each update, CVS records who made the changes and why the changes were made. Because CVS stores the most recent version of a file and the information needed to re-create all previous versions, it is possible to regenerate any version of a file.

A set of versions for several files may be grouped together to form a release. An entire release can be re-created from the change information stored with each file. Saving the changes for a file rather than saving a complete copy of the file generally conserves a lot of disk spacewell in excess of the space required to store each update in the CVS files themselves.

This section provides an overview of CVS and TkCVS. See the man page and the CVS home page for more information.

CVS: Concurrent Versions System

CVS treats collections of files as single units, making it easy to work on large projects and permitting multiple users to work on the same file. CVS also provides valuable self-documenting features for its utilities.

Builtin CVS Help

CVS uses a single utility, cvs, for all its functions. To display the instructions for getting help, use the help option:

$ cvs --help
Usage: cvs [cvs-options] command [command-options-and-arguments]
  where cvs-options are -q, -n, etc.
    (specify --help-options for a list of options)
  where command is add, admin, etc.
    (specify --help-commands for a list of commands
     or --help-synonyms for a list of command synonyms)
  where command-options-and-arguments depend on the specific command
    (specify -H followed by a command name for command-specific help)
  Specify --help to receive this message

The Concurrent Versions System (CVS) is a tool for version control.
For CVS updates and additional information, see
    the CVS home page at http://www.cvshome.org/ or
    Pascal Molli's CVS site at http://www.loria.fr/~molli/cvs-index.html

To get help with a cvs command, use the help option followed by the name of the command. The following example shows help information for the log command:

$ cvs --help log
Usage: cvs log [-lRhtNb] [-r[revisions]] [-d dates] [-s states]
    [-w[logins]] [files...]
        -l      Local directory only, no recursion.
        -R      Only print name of RCS file.
        -h      Only print header.
        -t      Only print header and descriptive text.
        -N      Do not list tags.
        -S      Do not print name/header if no revisions selected.
        -b      Only list revisions on the default branch.
        -r[revisions]   A comma-separated list of revisions to print:
           rev1:rev2   Between rev1 and rev2, including rev1 and rev2.
           rev1::rev2  Between rev1 and rev2, excluding rev1.
           rev:        rev and following revisions on the same branch.
           rev::       After rev on the same branch.
           :rev        rev and previous revisions on the same branch.
           ::rev       rev and previous revisions on the same branch.
           rev         Just rev.
           branch      All revisions on the branch.
           branch.     The last revision on the branch.
        -d dates        A semicolon-separated list of dates
                        (D1<D2 for range, D for latest before).
        -s states       Only list revisions with specified states.
        -w[logins]      Only list revisions checked in by specified logins.
(Specify the --help global option for a list of other help options)

Options for individual cvs commands (command options) go to the right of the individual command names. Options to the cvs utility itself, such as the help option to the log command, go to the left of all individual command names (that is, they follow the word cvs on the command line). The two types of options sometimes use the same letter yet may have an entirely different meaning.

How CVS Stores Revision Files

CVSROOT

With CVS, revision files are kept in a common area called a source repository. This area is identified by the value of the environment variable CVSROOT, which holds the absolute pathname of the repository. The system administrator can tell you what value to use for CVSROOT, or you can create your own private repository and have CVSROOT point to it.

The source repository is organized as a hierarchical collection of files and directories. CVS does not limit you to checking out one file at a time; you can check out an entire subdirectory containing many filestypically all the files for a particular project. A subdirectory of CVSROOT that can be checked out as a single unit is called a module. Several people can check out and simultaneously modify the files within a single module.

CVS users typically store the modules they are currently working on in a special directory. If you want to follow this practice, you must use cd to make that special directory the working directory before you check out a module. When you check out a module, CVS replicates the module's tree structure in the working directory. Multiple developers can therefore check out and edit CVS files simultaneously because the originals are retained in the source repository; the files in the repository undergo relatively infrequent modification in a controlled manner.

Basic CVS Commands

Although many cvs commands are available, knowledge of just a handful of commands allows a software developer to use CVS and to contribute changes to a module. A discussion of some useful commands follows. All examples assume that the appropriate modules have been installed in the CVS source repository. "Adding a Module to the Repository" (page 515) explains how to install a module.

Of the commands discussed in this section, cvs commit is the only one that changes the source repository. The other commands affect only the files in the working directory.

To simplify examples in the following sections, the pathname of the working directory is given by the variable CVSWORK; all modules can be assumed to be subdirectories of CVSWORK. Although this variable has no special meaning to CVS, you may find it helpful to define such a variable for your own work.

Checking Out Files from the Source Repository

To check out a module from the CVS source repository, use the cvs checkout command. The following example checks out the Project2 module, which consists of four source files. First use cd to make the directory you want the module copied into (CVSWORK in this case) the working directory. The cvs utility always copies into the working directory.

$ cd $CVSWORK
$ ls
Project1
$ cvs checkout Project2
cvs checkout: Updating Project2
U Project2/adata.h
U Project2/compute.c
U Project2/randomfile.h
U Project2/shuffle.c
$ ls
Project1 Project2
$ ls Project2
CVS adata.h compute.c randomfile.h shuffle.c

The name of the module, Project2, is given as an argument to cvs checkout. Because the Project2 directory does not already exist, cvs creates it in the working directory and places copies of all source files for the Project2 module into it. The name of the module and the name of the directory holding the module are the same. The checkout command preserves the tree structure of the cvs module, creating subdirectories as needed.

The second ls command after checkout reveals, in addition to the four source files for Project2, a directory named CVS. The CVS system uses this directory for administrative purposes; you do not normally access it.

Once you have your own copies of the source files, you can edit them as you see fit. You can change files within the module even if other developers are modifying the same files at the same time.

Making Your Changes Available to Others

To check in your changes so that other users have access to them, you need to run the cvs commit command. When you give this command, cvs prompts you to provide a brief log message describing the changes, unless you use the m option. With this option, cvs uses the string following the option as the log message. The file or files that you want to commit follow the optional log message on the command line:

$ cvs commit -m "function shuffle inserted" compute.c
cvs commit: Up-to-date check failed for 'compute.c'
cvs [commit aborted]: correct above errors first!

Here the cvs utility reports an error because the version of compute.c that you modified is not up-to-date. That is, a newer version of compute.c has been committed by someone else since you last checked it out of the source repository. After informing you of the problem, cvs exits without storing your changes in the source repository.

To make your version of compute.c current, you need to run the update command. A subsequent commit will then succeed, and your changes will apply to the latest revision in the source repository.

Updating Your Copies with Changes by Others

As the preceding example shows, CVS does not notify you when another developer checks in a new revision of a file after you have checked out your working copy. You learn this fact only when you attempt to commit your changes to the source repository. To incorporate up-to-date revisions of a CVS source file, use the cvs update command:

$ cvs update compute.c
RCS file: /usr/local/src/master/Project2/compute.c,v
retrieving revision 1.9
retrieving revision 1.10
Merging differences between 1.9 and 1.10 into compute.c
M compute.c

The changes made to the working copy of compute.c remain intact because the update command merges the latest revision in the source repository with the version specified on the update command line. The result of the merge is not always perfect, however. For this reason, the cvs update command informs you if it detects overlapping changes.

Adding New Files to the Repository

You can use the cvs add command to schedule new files to be added to the source repository as part of the module you are working on. Once you have used cd to make the directory containing the files the working directory, give the cvs add command, listing the files you want to add as arguments:

$ cd $CVSWORK/Project2
$ ls
CVS compute.c shuffle.c tabout2.c
adata.h randomfile.h tabout1.c
$ cvs add tabout[1-2].c
cvs add: scheduling file 'tabout1.c' for addition
cvs add: scheduling file 'tabout2.c' for addition
cvs add: use 'cvs commit' to add these files permanently

The add command marks the files tabout1.c and tabout2.c for entry into the repository. These files will not be available to others until you give a commit command. This staging allows you to prepare several files before others incorporate the changes into their working copies with the cvs update command.

Removing Files from the Repository

The cvs remove command records the fact that you wish to remove a file from the source repository. Like the add command, it does not affect the source repository. To delete a file from the repository, you must first delete your working copy of the file, as the following example shows:

$ cvs remove shuffle.c
cvs remove: file 'shuffle.c' still in working directory
cvs remove: 1 file exists; remove it first
$ rm shuffle.c
$ cvs remove shuffle.c
cvs remove: scheduling 'shuffle.c' for removal
cvs remove: use 'cvs commit' to remove this file permanently

After using rm to delete the working copy of shuffle.c, invoke a cvs remove command. Again, you must give the commit command before the file is actually removed from the source repository.

Other CVS Commands

Although the commands given earlier are sufficient for most work on a module, you may find some other commands to be useful.

Tagging a Release

You can apply a common label, or tag, to the files in a module as they currently exist. Once you have tagged files of a module, you can re-create them in exactly the same form even if they have been modified, added, or deleted since that time. This ability enables you to freeze a release yet allows development to continue on the next release.

$ cvs rtag Release_1 Project1
cvs rtag: Tagging Project1

Here the Project1 module has been tagged with the label Release_1. You can use this tag with the cvs export command to extract the files as they existed when they were frozen.

Extracting a Release

The cvs export command lets you extract files as they were frozen and tagged:

$ cvs export -r Release_1 -d R1 Project1
cvs export: Updating R1
U R1/scm.txt

This command works like the cvs checkout command but does not create the CVS support files. You must give either the r option to identify the release (as shown above) or a date with the D option. The d R1 option instructs cvs to place the files for the module into the directory R1 instead of using the module name as the directory.

Removing Working Files

When you are finished making changes to the files you have checked out of the repository, you may decide to remove your copy of the module from your working directory. One simple approach is to move into the working directory and recursively remove the module. For example, if you want to remove your working copy of Project2, you could use the following commands:

$ cd $CVSWORK
$ rm -rf Project2

The repository will not be affected by removing these files. However, if you had made changes to the files but had not yet committed those changes, the changes would be lost if you used this approach. The cvs release command is helpful in this situation:

$ cd $CVSWORK
$ cvs release -d Project2

The release command also removes the working files but first checks each one to see whether it has been marked for addition into the repository but has not yet been committed. If that is the case, the release command warns you and asks you to verify your intention to delete the file. You can fix the problem at this point if you like and redo the release command. The release command also warns you if the repository holds a newer version of the file than the one in your working directory. Thus you have the opportunity to update and commit your file before deleting it. (Without the d option, your working files will not be deleted but you will receive the same sequence of warning messages.)

Adding a Module to the Repository

The discussion of CVS to this point assumed that a module was already present in the CVS source repository. If you want to install a directory hierarchy as a new module in the repository or update an existing module with a new release that was developed elsewhere, go to the directory that holds the files for the project and run the cvs import command. The following example installs the files for Project1 in the source repository:

$ cvs import -m "My first project" Project1 ventag reltag

The m option allows you to enter a brief description of the module on the command line. Following the description is the directory or the pathname of the directory under CVSROOT that you want to hold the module. The last two fields are symbolic names for the vendor branch and the release. Although they are not significant here, they can be useful when releases of software are supplied by outside sources.

You can now use the cvs checkout command to check out the Project1 module:

$ cvs checkout Project1

CVS Administration

Before you install a CVS repository, think about how you would like to administer it. Many installations have a single repository where separate projects are kept as separate modules. You may choose to have more than one repository. The CVS system supports a single repository that is shared across several computer systems using NFS.

Inside a repository is a module named CVSROOT that contains administrative files (here CVSROOT is the name of a module and is different from the CVSROOT directory). Although the files in this module are not required to use CVS, they can simplify access to the repository.

Do not change any files in the CVSROOT module by editing them directly. Instead, check out the file you want to change, edit the checked-out copy, and then check it back in, just as you would with files in any other module in the repository. For example, to check out the modules file from the CVSROOT module, use the following command:

$ cvs checkout CVSROOT/modules

This command creates the directory CVSROOT in your working directory and places a checked-out copy of modules in that directory. After checking it out, you can edit the modules file in the CVSROOT directory:

$ cd CVSROOT
$ vim modules

After you edit the modules file, check it back into the repository:

$ cd ..
$ cvs checkin CVSROOT/modules

Of all the administrative files in the CVSROOT module, the modules file is the most important. You can use this file to attach symbolic names to modules in the repository, allow access to subdirectories of a module as if they were themselves modules, and specify actions to take when checking specific files in or out.

Most repositories start with a modules file that allows you to check out the modules file with the following command, instead of the one shown earlier:

$ cvs checkout modules

With the preceding command CVS creates a subdirectory named modules within the working directory, instead of one named CVSROOT. The modules file is then checked out into this directory.

The following example shows a modules file (the lines that start with # are comment lines and, along with blank lines, are ignored by CVS):

# Three different line formats are valid:
#       key      -a   aliases...
#       key [options] directory
#       key [options] directory files...
#
# Where "options" are composed of:
#       -i prog       Run "prog" on "cvs commit" from top-level of module.
#       -o prog       Run "prog" on "cvs checkout" of module.
#       -e prog       Run "prog" on "cvs export" of module.
#       -t prog       Run "prog" on "cvs rtag" of module.
#       -u prog       Run "prog" on "cvs update" of module.
#       -d dir       Place module in directory "dir" instead of module name.
#       -l            Top-level directory only -- do not recurse.
#
# NOTE: If you change any of the "Run" options above, you'll have to
# release and re-checkout any working directories of these modules.
#
# And "directory" is a path to a directory relative to $CVSROOT.
#
# The "-a" option specifies an alias. An alias is interpreted as if
# everything on the right of the "-a" had been typed on the command line.
#
# You can encode a module within a module by using the special '&'
# character to interpose another module into the current module. This
# can be useful for creating a module that consists of many directories
# spread out over the entire source repository.

# Convenient aliases
world      -a .
modules    -d . CVSROOT modules

# Add other modules here...
testgen    testgen
testdata1  testdata1
testdata2  testdata2
testdata3  testdata3
testdata4  testdata4
testcode   testgen/_code
cvs        cvs

The lines after the comment and blank lines define symbolic names for many modules. For example, the following line defines world to be an alias for the root of the CVS repository:

world     -a .

You can use such names in CVS commands as the names of modules. For example, the following command checks out the entire repository (probably not a good idea):

$ cvs checkout world

In the sample modules file, the administrative files have been given definitions that attach a symbolic name to the file.

Following the action is the name of the subdirectory in CVSROOT that holds any files associated with the symbolic name. Any remaining arguments on the line are the names of specific files within that directory.

The following line associates the modules module with the modules file within the CVSROOT directory:

modules     -d . CVSROOT modules

The d . option forces CVS to check out the file into the user's working directory instead of creating a new directory named modules. The preceding line allows the following command to find and check out the modules file:

$ cvs checkout modules

The last set of lines in the sample modules file associates symbolic module names with directories and files in the repository.

Using TkCVS

The cvs utility is useful enough that a graphical interface, TkCVS, has been written for it using the Tk extension to the Tcl programming language (tcl.sourceforge.net). It provides a convenient point-and-click interface to CVS (Figure 12-5). After you have downloaded and installed TkCVS (www.twobarleycorns.net/tkcvs.html), you can start the utility by opening the directory you installed it in and double-clicking the TkCVS icon.

Figure 12-5. The TkCVS utility

All operations are available through the pull-down menus found at the top of the window. Along the bottom of the window are buttons for accessing the most common actions. A description of the action bound to a button appears when you position the mouse pointer over a button.

In the middle of the window is a browse list. Move into a subdirectory by double-clicking the left mouse button while the mouse pointer is on the directory name in this list. Edit a file by double-clicking the filename. To select more than one file, hold down the left mouse button and drag the mouse pointer across several names. Clicking the right mouse button will mark all selected files. Some operations (such as viewing the revision log messages) will work on all marked files.

The Help pull-down menu in the upper-right corner of the window is an excellent way to learn how TkCVS works. For example, when you select the Help menu item CVS modules file..., an explanation of the lines that you can add to the CVS modules file to better support TkCVS appears in a window. If you choose not to add these lines to the modules file, some TkCVS commands, such as browsing the repository, may not display all available modules.


Chapter Summary

The operating system interface to C programs and a variety of software development tools render the Mac OS X system well suited to programming in C. The C libraries provide general-purpose C functions that make operating system services and other functionality available to C programmers. The standard C library libc is always accessible to C programs, and you can specify other libraries by using the l option to the gcc compiler.

You can write a C program by using a text editor, such as vim or emacs. C programs always have a function named main and often include several other functions. Preprocessor directives define symbolic constants and macros and instruct the preprocessor to include header files.

gcc

When you use gcc, it calls the C preprocessor followed by the C compiler and the assembler. The compiler creates assembly language code, which the assembler uses to create object modules. The linker combines these object modules into an executable file. You can use the Wall option to gcc to detect risky constructsones that are legal but suggest the possibility of later problems. Other options to gcc can help locate areas of your code that might not be portable.

gdb

Although using printf statements and the Wall option can help in tracking program bugs, it is a good practice to compile C programs routinely with the g option. Information that can be interpreted by gdb, a symbolic debugger, is then generated as part of the executable file. When you run a program under the control of gdb, you can specify points where gdb is to pause the program so that you can inquire about the values of variables, display the program stack, or use a wide range of commands to learn about many other aspects of the program's behavior.

make

The make utility uses a file named makefile (or Makefile) that documents the relationships among files. It determines which modules of a program are out-of-date and compiles files to keep all modules up-to-date. The dependency line, which specifies the exact dependency relationship between target and prerequisite files, is the key to the operation of a makefile. Following the dependency line are construction commands that can bring the target up-to-date. Implied dependencies, construction commands, and the make macro facility are available to simplify the writing of complex makefiles.

CVS

The Mac OS X system includes utilities that assist in keeping track of groups of files that undergo multiple revisions, often at the hands of multiple developers. These source code management systems include CVS (Concurrent Versions System). CVS is built on top of RCS but provides a much more extensive set of operations for managing directories of files that may be accessed and modified by many users. It is a good choice for managing large-scale projects and for maintaining software releases that are sent to and from other sites.


Exercises

1.

What function does every C program have? Why should you split large programs into several functions?

2.

What command could you use to compile prog.c and func.c into an executable named cprog?

3.

Show two ways to instruct the C preprocessor to include the header file /usr/include/math.h in your C program. Assuming that the declar.h header file is located in the subdirectory named headers of your home directory, describe two ways to instruct the C preprocessor to include this header file in your C program.

4.

How are the names of system libraries abbreviated on the gcc command line? Where does gcc search for libraries named in this manner? Describe how to specify your own library on the gcc command line.

5.

Write a makefile that reflects the following relationships:

  1. The C source files transactions.c and reports.c are compiled to produce an executable file named accts.

  2. Both transactions.c and reports.c include a header file named accts.h.

  3. The header file accts.h is composed of two other header files: trans.h and reps.h.

6.

If you retrieve version 4.1 of the file answer for editing and then attempt to retrieve the same version again, what will CVS do? Why is CVS set up this way?


Advanced Exercises

7.

Modify the badtabs.c program (page 497) so that it exits cleanly (with a specific return value). Compile the program and run it using gdb or another debugger. What values does the debugger report when the program finishes executing?

8.

For the makefile

$ cat Makefile
leads: menu.o users.o resellers.o prospects.o
             gcc -o leads menu.o users.o resellers.o prospects.o

menu.o: menu.h dialog.h inquiry.h

users.o: menu.h dialog.h

prospects.o: dialog.h

identify:

  1. Targets.

  2. Construction commands.

  3. Prerequisites.

9.

Refer to Makefile in exercise 8 to answer the following questions:

  1. If the target leads is up-to-date and you then change users.c, what happens when you run make again? Be specific.

  2. Rewrite the makefile to include the following macros:

    OBJECTS = menu.o users.o resellers.o prospects.o
HFILES = menu.h dialog.h

10.

Review the make info page to answer the following questions:

  1. What does the t option do?

  2. If you have files named makefile and Makefile in the working directory, how can you instruct make to use Makefile?

  3. Give two ways to define a variable so that you can use it inside a makefile.

11.

Refer to the makefile for compute on page 492.

  1. Suppose that a file in the working directory is named clean. What is the effect of giving the following command? Explain.

    $ make clean

  2. The discussion on page 491 states that the following command is not normally seen in makefiles:

    cat num.h table.h > form.h

    What is the effect of removing this construction command from the makefile while retaining the dependency line?

  3. The preceding construction command works only because the file form.h is made up of num.h and table.h. More often #include directives in the target define the dependencies. Suggest a more general technique that updates form.h whenever num.h or table.h has a more recent modification date.


Chapter 13. Programming the Bourne Again Shell

IN THIS CHAPTER

Control Structures | 524
File Descriptors | 555
Parameters and Variables | 558
Array Variables | 558
Locality of Variables | 560
Special Parameters | 562
Positional Parameters | 564
Builtin Commands | 570
Expressions | 584
Shell Programs | 593
A Recursive Shell Script | 593
The quiz Shell Script | 596

Chapter 5 introduced the shells and Chapter 8 went into detail about the Bourne Again Shell. This chapter introduces additional Bourne Again Shell commands, builtins, and concepts that carry shell programming to a point where it can be useful. The first part of this chapter covers programming control structures, which are also known as control flow constructs. These structures allow you to write scripts that can loop over command line arguments, make decisions based on the value of a variable, set up menus, and more. The Bourne Again Shell uses the same constructs found in such high-level programming languages as C.

The next part of this chapter discusses parameters and variables, going into detail about array variables, local versus global variables, special parameters, and positional parameters. The exploration of builtin commands covers type, which displays information about a command, and read, which allows you to accept user input in a shell script. The section on the exec builtin demonstrates how exec provides an efficient way to execute a command by replacing a process and explains how you can use it to redirect input and output from within a script. The next section covers the TRap builtin, which provides a way to detect and respond to operating system signals (such as that which is generated when you press CONTROL-C). The discussion of builtins concludes with a discussion of kill, which can abort a process, and getopts, which makes it easy to parse options for a shell script. (Table 13-6 on page 583 lists some of the more commonly used builtins.)

Next the chapter examines arithmetic and logical expressions and the operators that work with them. The final section walks through the design and implementation of two major shell scripts.

This chapter contains many examples of shell programs. Although they illustrate certain concepts, most use information from earlier examples as well. This overlap not only reinforces your overall knowledge of shell programming but also demonstrates how you can combine commands to solve complex tasks. Running, modifying, and experimenting with the examples in this book is a good way to become comfortable with the underlying concepts.

Tip: Do not name a shell script test

You can unwittingly create a problem if you give a shell script the name test because a Mac OS X utility has the same name. Depending on how the PATH variable is set up and how you call the program, you may run your script or the utility, leading to confusing results.

This chapter illustrates concepts with simple examples, which are followed by more complex ones in sections marked "Optional." The more complex scripts illustrate traditional shell programming practices and introduce some Mac OS X utilities often used in scripts. You can skip these sections without loss of continuity the first time you read the chapter. Return to them later when you feel comfortable with the basic concepts.


Control Structures

The control flow commands alter the order of execution of commands within a shell script. The TC Shell uses a different syntax for these commands (page 364) than the Bourne Again Shell does. Control structures include the if...then, for...in, while, until, and case statements. In addition, the break and continue statements work in conjunction with the control structures to alter the order of execution of commands within a script.

if...then

The if...then control structure has the following syntax:

if test-command
      then
          commands
fi

The bold words in the syntax description are the items you supply to cause the structure to have the desired effect. The nonbold words are the keywords the shell uses to identify the control structure.

test builtin

Figure 13-1 shows that the if statement tests the status returned by the test-command and transfers control based on this status. The end of the if structure is marked by a fi statement, (if spelled backward). The following script prompts for two words, reads them, and then uses an if structure to execute commands based on the result returned by the test builtin (tcsh uses the test utility) when it compares the two words. (See page 871 for information on the test utility, which is similar to the test builtin.) The test builtin returns a status of true if the two words are the same and false if they are not. Double quotation marks around $word1 and $word2 make sure that test works properly if you enter a string that contains a SPACE or other special character:

$ cat if1
echo -n "word 1: "
read word1
echo -n "word 2: "
read word2

if test "$word1" = "$word2"
    then
        echo "Match"
fi
echo "End of program."
$ if1
word 1: peach
word 2: peach
Match
End of program.

Figure 13-1. An if...then flowchart

In the preceding example the test-command is test "$word1" = "$word2". The test builtin returns a true status if its first and third arguments have the relationship specified by its second argument. If this command returns a true status (= 0), the shell executes the commands between the then and fi statements. If the command returns a false status (not = 0), the shell passes control to the statement following fi without executing the statements between then and fi. The effect of this if statement is to display Match if the two words are the same. The script always displays End of program.

Builtins

In the Bourne Again Shell, test is a builtinpart of the shell. It is also a stand-alone utility kept in /usr/bin/test. This chapter discusses and demonstrates many Bourne Again Shell builtins. Each bash builtin may or may not be a builtin in tcsh. You usually use the builtin version if it is available and the utility if it is not. Each version of a command may vary slightly from one shell to the next and from the utility to any of the shell builtins. See page 570 for more information on shell builtins.

Checking arguments

The next program uses an if structure at the beginning of a script to check that you have supplied at least one argument on the command line. The eq test operator compares two integers, where the $# special parameter (page 564) takes on the value of the number of command line arguments. This structure displays a message and exits from the script with an exit status of 1 if you do not supply at least one argument:

$ cat chkargs
if test $# -eq 0
    then
        echo "You must supply at least one argument."
        exit 1
fi
echo "Program running."
$ chkargs
You must supply at least one argument.
$ chkargs abc
Program running.

A test like the one shown in chkargs is a key component of any script that requires arguments. To prevent the user from receiving meaningless or confusing information from the script, the script needs to check whether the user has supplied the appropriate arguments. Sometimes the script simply tests whether arguments exist (as in chkargs). Other scripts test for a specific number or specific kinds of arguments.

You can use test to ask a question about the status of a file argument or the relationship between two file arguments. After verifying that at least one argument has been given on the command line, the following script tests whether the argument is the name of an ordinary file (not a directory or other type of file) in the working directory. The test builtin with the f option and the first command line argument ($1) check the file:

$ cat is_ordfile
if test $# -eq 0
    then
        echo "You must supply at least one argument."
        exit 1
fi
if test -f "$1"
    then
        echo "$1 is an ordinary file in the working directory"
    else
        echo "$1 is NOT an ordinary file in the working directory"
fi

You can test many other characteristics of a file with test and various options. Table 13-1 lists some of these options.

Table 13-1. Options to the test builtin

Option

Tests file to see if it

d

Exists and is a directory file

e

Exists

f

Exists and is an ordinary file (not a directory)

r

Exists and is readable

s

Exists and has a size greater than 0 bytes

w

Exists and is writable

x

Exists and is executable

Other test options provide ways to test relationships between two files, such as whether one file is newer than another. Refer to later examples in this chapter and to test on page 871 for more detailed information.

Tip: Always test the arguments

To keep the examples in this book short and focused on specific concepts, the code to verify arguments is often omitted or abbreviated. It is a good practice to test arguments in shell programs that other people will use. Doing so results in scripts that are easier to run and debug.

[] is a synonym for test

The following exampleanother version of chkargschecks for arguments in a way that is more traditional for Mac OS X shell scripts. The example uses the bracket ([]) synonym for test. Rather than using the word test in scripts, you can surround the arguments to test with brackets. The brackets must be surrounded by whitespace (SPACEs or TABs).

$ cat chkargs2
if [ $# -eq 0 ]
    then
        echo "Usage: chkargs2 argument..." 1>&2
        exit 1
fi
echo "Program running."
exit 0
$ chkargs2
Usage: chkargs2 arguments
$ chkargs2 abc
Program running.

Usage message

The error message that chkargs2 displays is called a usage message and uses the 1>&2 notation to redirect its output to standard error (page 262). After issuing the usage message, chkargs2 exits with an exit status of 1, indicating that an error has occurred. The exit 0 command at the end of the script causes chkargs2 to exit with a 0 status after the program runs without an error. The Bourne Again Shell returns a 0 status if you omit the status code.

The usage message is commonly employed to specify the type and number of arguments the script takes. Many UNIX utilities provide usage messages similar to the one in chkargs2. If you call a utility or other program with the wrong number or kind of arguments, you will often see a usage message. Following is the usage message that cp displays when you call it without any arguments:

$ cp
cp: missing file argument
Try 'cp --help' for more information.

if...then...else

The introduction of an else statement turns the if structure into the two-way branch shown in Figure 13-2. The if...then...else control structure (available in tcsh with a slightly different syntax) has the following syntax:

if test-command
      then
          commands
      else
          commands
fi

Figure 13-2. An if...then...else flowchart

Because a semicolon (;) ends a command just as a NEWLINE does, you can place then on the same line as if by preceding it with a semicolon. (Because if and then are separate builtins, they require a command separator between them; a semicolon and NEWLINE work equally well.) Some people prefer this notation for aesthetic reasons, while others like it because it saves space:

if test-command; then
          commands
      else
          commands
fi

If the test-command returns a true status, the if structure executes the commands between the then and else statements and then diverts control to the statement following fi. If the test-command returns a false status, the if structure executes the commands following the else statement.

When you run the next script, named out, with arguments that are filenames, it displays the files on the terminal. If the first argument is v (called an option in this case), out uses less (page 42) to display the files one page at a time. After determining that it was called with at least one argument, out tests its first argument to see whether it is v. If the result of the test is true (if the first argument is v), out uses the shift builtin to shift the arguments to get rid of the v and displays the files using less. If the result of the test is false (if the first argument is not v), the script uses cat to display the files:

$ cat out
if [ $# -eq 0 ]
    then
        echo "Usage: out [-v] filenames..." 1>&2
        exit 1
fi
if [ "$1" = "-v" ]
    then
        shift
        less -- "$@"
    else
        cat -- "$@"
fi

Optional

In out the -- argument to cat and less tells these utilities that no more options follow on the command line and not to consider leading hyphens (-) in the following list as indicating options. Thus -- allows you to view a file with a name that starts with a hyphen. Although not common, filenames beginning with a hyphen do occasionally occur. (You can create such a file by using the command cat > -fname.) The -- argument works with all utilities that use the getopts builtin (page 581) to parse their options; it does not work with a few utilities. This argument is particularly useful when used in conjunction with rm to remove a file whose name starts with a hyphen (rm -- -fname), including any that you create while experimenting with the -- argument.

if...then...elif

The if...then...elif control structure (Figure 13-3; not available in tcsh) has the following syntax:

if test-command
      then
          commands
      elif test-command
          then
              commands
...
      else
          commands
fi

Figure 13-3. An if...then...elif flowchart

The elif statement combines the else statement and the if statement and allows you to construct a nested set of if...then...else structures (Figure 13-3). The difference between the else statement and the elif statement is that each else statement must be paired with a fi statement, whereas multiple nested elif statements require only a single closing fi statement.

The following example shows an if...then...elif control structure. This shell script compares three words that the user enters. The first if statement uses the Boolean operator AND (a) as an argument to test. The test builtin returns a true status only if the first and second logical comparisons are true (that is, if word1 matches word2 and word2 matches word3). If test returns a true status, the script executes the command following the next then statement, passes control to the statement following fi, and terminates:

$ cat if3
echo -n "word 1: "
read word1
echo -n "word 2: "
read word2
echo -n "word 3: "
read word3
if [ "$word1" = "$word2" -a "$word2" = "$word3" ]
    then
        echo "Match: words 1, 2, & 3"
    elif [ "$word1" = "$word2" ]
    then
        echo "Match: words 1 & 2"
    elif [ "$word1" = "$word3" ]
    then
        echo "Match: words 1 & 3"
    elif [ "$word2" = "$word3" ]
    then
        echo "Match: words 2 & 3"
    else
        echo "No match"
fi
$ if3
word 1: apple
word 2: orange
word 3: pear
No match
$ if3
word 1: apple
word 2: orange
word 3: apple
Match: words 1 & 3
$ if3
word 1: apple
word 2: apple
word 3: apple
Match: words 1, 2, & 3

If the three words are not the same, the structure passes control to the first elif, which begins a series of tests to see if any pair of words is the same. As the nesting continues, if any one of the if statements is satisfied, the structure passes control to the next then statement and subsequently to the statement following fi. Each time an elif statement is not satisfied, the structure passes control to the next elif statement. The double quotation marks around the arguments to echo that contain ampersands (&) prevent the shell from interpreting the ampersands as special characters.

Optional: The lnks Script

The following script, named lnks, demonstrates the if...then and if...then...elif control structures. This script finds hard links to its first argument, a filename. If you provide the name of a directory as the second argument, lnks searches for links in that directory and all subdirectories. If you do not specify a directory, lnks searches the working directory and its subdirectories. This script does not locate symbolic links.

$ cat lnks
#!/bin/bash
# Identify links to a file
# Usage: lnks file [directory]

if [ $# -eq 0 -o $# -gt 2 ]; then
    echo "Usage: lnks file [directory]" 1>&2
    exit 1
fi
if [ -d "$1" ]; then
    echo "First argument cannot be a directory." 1>&2
    echo "Usage: lnks file [directory]" 1>&2
    exit 1
else
    file="$1"
fi
if [ $# -eq 1 ]; then
       directory="."
   elif [ -d "$2" ]; then
       directory="$2"
   else
       echo "Optional second argument must be a directory." 1>&2
       echo "Usage: lnks file [directory]" 1>&2
       exit 1
fi

# Check that file exists and is an ordinary file:
if [ ! -f "$file" ]; then
    echo "lnks: $file not found or special file" 1>&2
    exit 1
fi
# Check link count on file
set -- $(ls -l "$file")

linkcnt=$2
if [ "$linkcnt" -eq 1 ]; then
    echo "lnks: no other hard links to $file" 1>&2
    exit 0
fi

# Get the inode of the given file
set $(ls -i "$file")

inode=$1

# Find and print the files with that inode number
echo "lnks: using find to search for links..." 1>&2
find -x "$directory" -inum $inode -print

Alex has a file named letter in his home directory. He wants to find links to this file in his and other users' home directory file trees. In the following example, Alex calls lnks from his home directory to perform the search. The second argument to lnks, /Users, is the pathname of the directory he wants to start the search in. The lnks script reports that /Users/alex/letter and /Users/jenny/draft are links to the same file:

$ lnks letter /Users
lnks: using find to search for links...
/Users/alex/letter
/Users/jenny/draft

In addition to the if...then...elif control structure, lnks introduces other features that are commonly used in shell programs. The following discussion describes lnks section by section.

Specify the shell

The first line of the lnks script uses #! (page 267) to specify the shell that will execute the script:

#!/bin/bash

In this chapter the #! notation appears only in more complex examples. It ensures that the proper shell executes the script, even when the user is running a different shell or the script is called from another shell script.

Comments

The second and third lines of lnks are comments; the shell ignores the text that follows a pound sign up to the next NEWLINE character. These comments in lnks briefly identify what the file does and how to use it:

# Identify links to a file
# Usage: lnks file [directory]

Usage messages

The first if statement tests whether lnks was called with zero arguments or more than two arguments:

if [ $# -eq 0 -o $# -gt 2 ]; then
    echo "Usage: lnks file [directory]" 1>&2
    exit 1
fi

If either of these conditions is true, lnks sends a usage message to standard error and exits with a status of 1. The double quotation marks around the usage message prevent the shell from interpreting the brackets as special characters. The brackets in the usage message indicate that the directory argument is optional.

The second if statement tests whether the first command line argument ($1) is a directory (the d argument to test returns a true value if the file exists and is a directory):

if [ -d "$1" ]; then
    echo "First argument cannot be a directory." 1>&2
    echo "Usage: lnks file [directory]" 1>&2
    exit 1
else
    file="$1"
fi

If the first argument is a directory, lnks displays a usage message and exits. If it is not a directory, lnks saves the value of $1 in the file variable because later in the script set resets the command line arguments. If the value of $1 is not saved before the set command is issued, its value will be lost.

Test the arguments

The next section of lnks is an if...then...elif statement:

if [ $# -eq 1 ]; then
        directory="."
    elif [ -d "$2" ]; then
        directory="$2"
    else
        echo "Optional second argument must be a directory." 1>&2
        echo "Usage: lnks file [directory]" 1>&2
        exit 1
fi

The first test-command determines whether the user specified a single argument on the command line. If the test-command returns 0 (true), the user-created variable named directory is assigned the value of the working directory (.). If the test-command returns false, the elif statement tests whether the second argument is a directory. If it is a directory, the directory variable is set equal to the second command line argument, $2. If $2 is not a directory, lnks sends a usage message to standard error and exits with a status of 1.

The next if statement in lnks tests whether $file does not exist. This test keeps lnks from wasting time looking for links to a nonexistent file.

The test builtin with the three arguments !, f, and $file evaluates to true if the file $file does not exist:

[ ! -f "$file" ]

The ! operator preceding the f argument to test negates its result, yielding false if the file $file does exist and is an ordinary file.

Next lnks uses set and ls l to check the number of links $file has:

# Check link count on file
set -- $(ls -l "$file")

linkcnt=$2
if [ "$linkcnt" -eq 1 ]; then
    echo "lnks: no other hard links to $file" 1>&2
    exit 0
fi

The set builtin uses command substitution (page 327) to set the positional parameters to the output of ls l. The second field in this output is the link count, so the user-created variable linkcnt is set equal to $2. The used with set prevents set from interpreting as an option the first argument produced by ls l (the first argument is the access permissions for the file and typically begins with ). The if statement checks whether $linkcnt is equal to 1; if it is, lnks displays a message and exits. Although this message is not truly an error message, it is redirected to standard error. The way lnks has been written, all informational messages are sent to standard error. Only the final product of lnksthe pathnames of links to the specified fileis sent to standard output, so you can redirect the output as you please.

If the link count is greater than one, lnks goes on to identify the inode (page 937) for $file. As explained on page 99, comparing the inodes associated with filenames is a good way to determine whether the filenames are links to the same file. (On an HFS+ filesystem, the underlying data structures are different, but the system behaves comparably.) The lnks script uses set to set the positional parameters to the output of ls i. The first argument to set is the inode number for the file, so the user-created variable named inode is assigned the value of $1:

# Get the inode of the given file
set $(ls -i "$file")
inode=$1

Finally lnks uses the find utility (page 728) to search for files having inode numbers that match $inode:

# Find and print the files with that inode number
echo "lnks: using find to search for links..." 1>&2
find -x "$directory" -inum $inode -print

The find utility searches for files that meet the criteria specified by its arguments, beginning its search with the directory specified by its first argument ($directory) and searching all subdirectories. The remaining arguments specify that the filenames of files having inodes matching $inode should be sent to standard output. Because files in different filesystems can have the same inode number and not be linked, find must search only directories in the same filesystem as $directory. The x argument prevents find from searching directories on other filesystems. Refer to page 102 for more information about filesystems and links.

The echo command preceding the find command in lnks, which tells the user that find is running, is included because find frequently takes a long time to run. Because lnks does not include a final exit statement, the exit status of lnks is that of the last command it runs, find.

DEBUGGING SHELL SCRIPTS

When you are writing a script such as lnks, it is easy to make mistakes. You can use the shell's x option to help debug a script. This option causes the shell to display each command before it runs the command. Tracing a script's execution in this way can give you information about where a problem lies.

You can run lnks as in the previous example and cause the shell to display each command before it is executed. Either set the x option for the current shell (set x) so that all scripts display commands as they are run or use the x option to affect only the shell that is running the script called by the command line.

$ bash -x lnks letter /Users
+ '[' 2 -eq 0 -o 2 -gt 2 ']'
+ '[' -d letter ']'
+ file=letter
+ '[' 2 -eq 1 ']'
+ '[' -d /Users ']'
+ directory=/Users
+ '[' '!' -f letter ']'
...

PS4

Each command that the script executes is preceded by the value of the PS4 variablea plus sign (+) by default, so you can distinguish debugging output from script-produced output. You must export PS4 if you set it in the shell that calls the script. The next command sets PS4 to >>>> followed by a SPACE and exports it:

$ export PS4='>>>> '

You can also set the x option of the shell running the script by putting the following set command at the top of the script:

set -x

Put set x anywhere in the script you want to turn debugging on. Turn the debugging option off with a plus sign.

set +x

The set o xtrace and set +o xtrace commands do the same things as set x and set +x, respectively.

for...in

The for...in control structure (Figure 13-4 [tcsh uses foreach]) has the following syntax:

for loop-index in argument-list
do
    commands
done

Figure 13-4. A for...in flowchart

The for...in structure assigns the value of the first argument in the argument-list to the loop-index and executes the commands between the do and done statements. The do and done statements mark the beginning and end of the for loop.

After it passes control to the done statement, the structure assigns the value of the second argument in the argument-list to the loop-index and repeats the commands. The structure repeats the commands between the do and done statements one time for each argument in the argument-list. When the structure exhausts the argument-list, it passes control to the statement following done.

The following for...in structure assigns apples to the user-created variable fruit and then displays the value of fruit, which is apples. Next the structure assigns oranges to fruit and repeats the process. When it exhausts the argument list, the structure transfers control to the statement following done, which displays a message.

$ cat fruit
for fruit in apples oranges pears bananas
do
   echo "$fruit"
done
echo "Task complete."

$ fruit
apples
oranges
pears
bananas
Task complete.

The next script lists the names of the directory files in the working directory by looping over all the files, using test to determine which files are directories:

$ cat dirfiles
for i in *
do
    if [ -d "$i" ]
       then
           echo "$i"
    fi
done

The ambiguous file reference character * matches the names of all files (except invisible files) in the working directory. Prior to executing the for loop, the shell expands the * and uses the resulting list to assign successive values to the index variable i.

for

The for control structure (not available in tcsh) has the following syntax:

for loop-index
do
    commands
done

In the for structure the loop-index takes on the value of each of the command line arguments, one at a time. It is the same as the for...in structure (Figure 13-4) except for where it gets values for the loop-index. The for structure performs a sequence of commands, usually involving each argument in turn.

The following shell script shows a for structure displaying each command line argument. The first line of the script, for arg, implies for arg in "$@", where the shell expands "$@" into a list of quoted command line arguments "$1" "$2" "$3" and so on. The balance of the script corresponds to the for...in structure.

$ cat for_test
for arg
do
    echo "$arg"
done
$ for_test candy gum chocolate
candy
gum
chocolate

Optional: The whos Script

The following script, named whos, demonstrates the usefulness of the implied "$@" in the for structure. You give whos one or more user or login names as arguments, and whos displays information about the users. The whos script gets the information it displays from the first and eighth fields in the passwd database. The first field always contains a username, and the eighth field typically contains the user's full name. You can provide a login name as an argument to whos to identify the user's name or provide a name as an argument to identify the username. The whos script is similar to the finger utility, although whos delivers less information.

$ cat whos
#!/bin/bash
# adapted from finger.sh by Lee Sailer
# UNIX/WORLD, III:11, p. 67, Fig. 2

if [ $# -eq 0 ]
    then
        echo "Usage: whos id..." 1>&2
        exit 1
fi
for id
do
    nidump passwd /
    awk -F: '{print $1, $8}' |
    grep -i "$id"
done

Below whos identifies the user whose username is chas and the user whose name is Marilou Smith:

$ whos chas "Marilou Smith"
chas Charles Casey
msmith Marilou Smith

Use of "$@"

The whos script uses a for statement to loop through the command line arguments. In this script the implied use of "$@" in the for loop is particularly beneficial because it causes the for loop to treat an argument that contains a SPACE as a single argument. This example quotes Marilou Smith, which causes the shell to pass it to the script as a single argument. Then the implied "$@" in the for statement causes the shell to regenerate the quoted argument Marilou Smith so that it is again treated as a single argument.

awk

For each command line argument, whos searches the passwd database. The nidump utility (NetInfo dump, [page 796]) displays the contents of the database. Inside the for loop the awk utility (page 609) extracts the first ($1) and eighth ($8) fields from the lines in the passwd database. The F: option causes awk to use a colon (:) as a field separator when it reads the passwd database, allowing it to break each line into fields. The awk command sets and uses the $1 and $8 arguments; they are included within single quotation marks and are not interpreted by the shell. Do not confuse these arguments with positional parameters, which correspond to command line arguments. The first and eighth fields are sent to grep (page 751) via a pipe. The grep utility searches for $id (which has taken on the value of a command line argument) in its input. The i option causes grep to ignore case as it searches; grep displays each line in its input that contains $id.

| at the end of a line

An interesting syntactical exception that bash gives the pipe symbol (|) appears on the line with the awk command: You do not have to quote a NEWLINE that immediately follows a pipe symbol (that is, a pipe symbol that is the last thing on a line) to keep the NEWLINE from executing a command. Try giving the command who | and pressing RETURN. The shell (not tcsh) displays a secondary prompt. If you then enter sort followed by another RETURN, you see a sorted who list. The pipe works even though a NEWLINE follows the pipe symbol.

while

The while control structure (not available in tcsh) has the following syntax:

while test-command
do
    commands
done

As long as the test-command (Figure 13-5) returns a true exit status, the while structure continues to execute the series of commands delimited by the do and done statements. Before each loop through the commands, the structure executes the test-command. When the exit status of the test-command is false, the structure passes control to the statement after the done statement.

Figure 13-5. A while flowchart

test builtin

The following shell script first initializes the number variable to zero. The test builtin then determines whether number is less than 10. The script uses test with the lt argument to perform a numerical test. For numerical comparisons, you must use ne (not equal), eq (equal), gt (greater than), ge (greater than or equal to), lt (less than), or le (less than or equal to). For string comparisons use = (equal) or != (not equal) when you are working with test. In this example, test has an exit status of 0 (true) as long as number is less than 10. As long as test returns true, the structure executes the commands between the do and done statements. See page 871 for information on the test utility, which is very similar to the test builtin.

$ cat count
#!/bin/bash
number=0
while [ "$number" -lt 10 ]
    do
       echo -n "$number"
       ((number +=1))
    done
echo
$ count
0123456789
$

The echo command following do displays number. The n prevents echo from issuing a NEWLINE following its output. The next command uses arithmetic evaluation [((...)); page 585)] to increment the value of number by 1. The done statement terminates the loop and returns control to the while statement to start the loop over again. The final echo causes count to send a NEWLINE character to standard output, so that the next prompt occurs in the leftmost column on the display (rather than immediately following 9).

until

The until (not available in tcsh) and while (available in tcsh with a slightly different syntax) structures are very similar, differing only in the sense of the test performed at the top of the loop. Figure 13-6 shows that until continues to loop until the test-command returns a true exit status. The while structure loops while the test-command continues to return a true or nonerror condition. The until control structure has the following syntax:

until test-command
do
    commands
done

Figure 13-6. An until flowchart

The following script demonstrates an until structure that includes read. When the user enters the correct string of characters, the test-command is satisfied and the structure passes control out of the loop.

$ cat until1
secretname=jenny
name=noname
echo "Try to guess the secret name!"
echo
until [ "$name" = "$secretname" ]
do
    echo -n "Your guess: "
    read name
done
echo "Very good."

$ until1
Try to guess the secret name!

Your guess: helen
Your guess: barbara
Your guess: rachael
Your guess: jenny
Very good

The following locktty script is similar to the lock command on Berkeley UNIX. The script prompts you for a key (password) and uses an until control structure to lock the terminal. The until statement causes the system to ignore any characters typed at the keyboard until the user types in the key on a line by itself, which unlocks the terminal. The locktty script can keep people from using your terminal while you are away from it for short periods of time. It saves you from having to log out if you are concerned about other users using your login.

$ cat locktty
#! /bin/bash
# UNIX/WORLD, III:4

trap '' 1 2 3 18
stty -echo
echo -n "Key: "
read key_1
echo
echo -n "Again: "
read key_2
echo
key_3=
if [ "$key_1" = "$key_2" ]
    then
        tput clear
        until [ "$key_3" = "$key_2" ]
        do
            read key_3
        done
    else
        echo "locktty: keys do not match" 1>&2
fi
stty echo

Tip: Forget your password for locktty?

If you forget your key (password), you will need to log in from another (virtual) terminal and kill the process running locktty.

trap builtin

The trap builtin (page 577; not available in tcsh) at the beginning of the locktty script stops a user from being able to terminate the script by sending it a signal (for example, by pressing the interrupt key). Trapping signal 18 means that no one can use CONTROLZ (job control, a stop from a tty) to defeat the lock. (See Table 13-5 on page 577 for a list of signals.) The stty echo command (page 854) causes the terminal not to display characters typed at the keyboard, thereby preventing the key that the user enters from appearing on the screen. After turning off keyboard echo, the script prompts the user for a key, reads it into the user-created variable key_1, prompts the user to enter the same key again, and saves it in key_2. The statement key_3= creates a variable with a NULL value. If key_1 and key_2 match, locktty clears the screen (with the tput command) and starts an until loop. The until loop keeps attempting to read from the terminal and assigning the input to the key_3 variable. Once the user types in a string that matches one of the original keys (key_2), the until loop terminates and keyboard echo is turned on again.

break And continue

You can interrupt a for, while, or until loop by using a break or continue statement. The break statement transfers control to the statement after the done statement, which terminates execution of the loop. The continue command transfers control to the done statement, which continues execution of the loop.

The following script demonstrates the use of these two statements. The for...in structure loops through the values 110. The first if statement executes its commands when the value of the index is less than or equal to 3 ($index le 3). The second if statement executes its commands when the value of the index is greater than or equal to 8 ($index ge 8). In between the two ifs, echo displays the value of the index. For all values up to and including 3, the first if statement displays continue and executes a continue statement that skips echo $index and the second if statement and continues with the next for statement. For the value of 8, the second if statement displays break and executes a break statement that exits from the for loop:

$ cat brk
for index in 1 2 3 4 5 6 7 8 9 10
    do
       if [ $index -le 3 ] ; then
           echo "continue"
           continue
       fi
    echo $index
    if [ $index -ge 8 ] ; then
       echo "break"
       break
    fi
done
$ brk
continue
continue
continue
4
5
6
7
8
break

case

The case structure (Figure 13-7) is a multiple-branch decision mechanism. The path taken through the structure depends on a match or lack of a match between the test-string and one of the patterns. The case control structure (tcsh uses switch) has the following syntax:

case test-string in
    pattern-1)
        commands-1
        ;;
    pattern-2)
        commands-2
        ;;
    pattern-3)
        commands-3
        ;;
...
esac

Figure 13-7. A case flowchart

The following case structure examines the character that the user enters as the test-string. This value is held in the variable letter. If the test-string has a value of A, the structure executes the command following the pattern A. The right parenthesis is part of the case control structure, not part of the pattern. If the test-string has a value of B or C, the structure executes the command following the matching pattern. The asterisk (*) indicates any string of characters and serves as a catchall in case there is no match. If no pattern matches the test-string and if there is no catchall (*) pattern, control passes to the command following the esac statement, without the case structure taking any action.

$ cat case1
echo -n "Enter A, B, or C: "
read letter
case "$letter" in
    A)
       echo "You entered A"
       ;;
    B)
       echo "You entered B"
       ;;
    C)
       echo "You entered C"
       ;;
    *)
       echo "You did not enter A, B, or C"
       ;;
esac
$ case1
Enter A, B, or C: B
You entered B

The next execution of case1 shows the user entering a lowercase b. Because the test-string b does not match the uppercase B pattern (or any other pattern in the case statement), the program executes the commands following the catchall pattern and displays a message:

$ case1
Enter A, B, or C: b
You did not enter A, B, or C

The pattern in the case structure is analogous to an ambiguous file reference. It can include any of the special characters and strings shown in Table 13-2.

Table 13-2. Patterns

Pattern

Function

*

Matches any string of characters. Use for the default case.

?

Matches any single character.

[...]

Defines a character class. Any characters enclosed within brackets are tried, one at a time, in an attempt to match a single character. A hyphen between two characters specifies a range of characters.

|

Separates alternative choices that satisfy a particular branch of the case structure.

The next script accepts both uppercase and lowercase letters:

$ cat case2
echo -n "Enter A, B, or C: "
read letter
case "$letter" in
    a|A)
       echo "You entered A"
       ;;
    b|B)
       echo "You entered B"
       ;;
    c|C)
       echo "You entered C"
       ;;
    *)
       echo "You did not enter A, B, or C"
       ;;
esac
$ case2
Enter A, B, or C: b
You entered B

Optional

The following example shows how you can use the case structure to create a simple menu. The command_menu script uses echo to present menu items and prompt the user for a selection. (The select control structure [page 551] makes it much easier to code a menu.) The case structure then executes the appropriate utility depending on the user's selection.

$ cat command_menu
#!/bin/bash
# menu interface to simple commands

echo -e "\n      COMMAND MENU\n"
echo "  a.  Current date and time"
echo "  b.  Users currently logged in"
echo "  c.  Name of the working directory"
echo -e "  d.  Contents of the working directory\n"
echo -n "Enter a, b, c, or d: "
read answer
echo
#
case "$answer" in
    a)
       date
       ;;
    b)
       who
       ;;
    c)
       pwd
       ;;
    d)
       ls
       ;;
    *)
       echo "There is no selection: $answer"
       ;;
esac
$ command_menu

           COMMAND MENU

    a.  Current date and time
    b.  Users currently logged in
    c.  Name of the working directory
    d.  Contents of the working directory

Enter a, b, c, or d: a

Wed Jan 5 12:31:12 PST 2005

echo e

The e option causes echo to interpret \n as a NEWLINE character. If you do not include this option, echo does not output the extra blank lines that make the menu easy to read but instead outputs the (literal) two-character sequence \n. The e option causes echo to interpret several other backslash-quoted characters (Table 13-3). Remember to quote (i.e., place double quotation marks around the string) the backslash-quoted character so that the shell does not interpret it but passes the backslash and the character to echo. See xpg_echo (page 321) for a way to avoid using the e option.

Table 13-3. Special characters in echo (must use e)

Quoted character

echo displays

\a

Alert (bell)

\b

BACKSPACE

\c

Suppress trailing NEWLINE

\f

FORMFEED

\n

NEWLINE

\r

RETURN

\t

Horizontal TAB

\v

Vertical TAB

\\

Backslash

\nnn

The character with the ASCII octal code nnn; if nnn is not valid, echo displays the string literally

You can also use the case control structure to take various actions in a script, depending on how many arguments the script is called with. The following script, named safedit, uses a case structure that branches based on the number of command line arguments ($#). It saves a backup copy of a file you are editing with vim.

$ cat safedit
#!/bin/bash
# UNIX/WORLD, IV:11

PATH=/bin:/usr/bin
script=$(basename $0)
case $# in

    0)
       vim
       exit 0
       ;;

    1)
       if [ ! -f "$1" ]
           then
               vim "$1"
               exit 0
           fi
       if [ ! -r "$1" -o ! -w "$1" ]
           then
               echo "$script: check permissions on $1" 1>&2
               exit 1
           else
               editfile=$1
           fi
       if [ ! -w "." ]
           then
               echo "$script: backup cannot be " \
                   "created in the working directory" 1>&2
               exit 1
           fi
        ;;
    *)
        echo "Usage: $script [file-to-edit]" 1>&2
        exit 1
        ;;
esac
tempfile=/tmp/$$.$script
cp $editfile $tempfile
if vim $editfile
   then
       mv $tempfile bak.$(basename $editfile)
       echo "$script: backup file created"
   else
       mv $tempfile editerr
       echo "$script: edit error--copy of " \
           "original file is in editerr" 1>&2
fi

If you call safedit without any arguments, the case structure executes its first branch and calls vim without a filename argument. Because an existing file is not being edited, safedit does not create a backup file. (See the :w command on page 158 for an explanation of how to exit from vim when you have called it without a filename.) If you call safedit with one argument, it runs the commands in the second branch of the case structure and verifies that the file specified by $1 does not yet exist or is the name of a file for which the user has read and write permission. The safedit script also verifies that the user has write permission for the working directory. If the user calls safedit with more than one argument, the third branch of the case structure presents a usage message and exits with a status of 1.

Set PATH

In addition to using a case structure for branching based on the number of command line arguments, the safedit script introduces several other features. First, at the beginning of the script, the PATH variable is set to search /bin and /usr/bin. Setting PATH in this way ensures that the commands executed by the script are standard utilities, which are kept in those directories. By setting PATH inside a script, you can avoid the problems that might occur if users have set PATH to search their own directories first and have scripts or programs with the same names as the utilities the script calls. You can also include absolute pathnames within a script to achieve this end, but this practice can make a script less portable.

Name of the program

In a second safedit feature, the following line creates a variable named script and assigns the simple filename of the script to it:

script=$(basename $0)

The basename utility sends the simple filename component of its argument to standard output, which is assigned to the script variable, using command substitution. The $0 holds the command the script was called with (page 565). No matter which of the following commands the user calls the script with, the output of basename is the simple filename safedit:

$ /Users/alex/bin/safedit memo
$ ./safedit memo
$ safedit memo

After the script variable is set, it replaces the filename of the script in usage and error messages. By using a variable that is derived from the command that invoked the script rather than a filename that is hardcoded into the script, you can create links to the script or rename it, and the usage and error messages will still provide accurate information.

Naming temporary files

A third significant feature of safedit relates to the use of the $$ variable in the name of a temporary file. The statement following the esac statement creates and assigns a value to the tempfile variable. This variable contains the name of a temporary file that is stored in the /tmp directory, as are many temporary files. The temporary filename begins with the PID number of the shell and ends with the name of the script. Use of the PID number ensures that the filename is unique, and safedit will not attempt to overwrite an existing file, as might happen if two people were using safedit at the same time. The name of the script is appended so that, should the file be left in /tmp for some reason, you can figure out where it came from.

The PID number is used in front ofrather than after$script in the filename because of the 14-character limit placed on filenames by some older versions of UNIX. Mac OS X systems do not have this limitation. Because the PID number ensures the uniqueness of the filename, it is placed first so that it cannot be truncated. (If the $script component is truncated, the filename is still unique.) For the same reason, when a backup file is created inside the if control structure a few lines down in the script, the filename is composed of the string bak. followed by the name of the file being edited. On an older system, if bak were used as a suffix rather than a prefix and the original filename were 14 characters long, .bak might be lost and the original file would be overwritten. The basename utility extracts the simple filename of $editfile before it is prefixed with bak.

Fourth, safedit uses an unusual test-command in the if structure: vim $editfile. The test-command calls vim to edit $editfile. When you finish editing the file and exit from vim, vim returns an exit code. The if control structure uses that exit code to determine which branch to take. If the editing session completed successfully, vim returns 0 and the statements following the then statement are executed. If vim does not terminate normally (as would occur if the user killed [page 761] the vim process), vim returns a nonzero exit status and the script executes the statements following else.

select

The select control structure (not available in tcsh) is based on the one found in the Korn Shell. It displays a menu, assigns a value to a variable based on the user's choice of items, and executes a series of commands. The select control structure has the following syntax:

select varname [in arg ... ]
do
    commands
done

The select structure displays a menu of the arg items. If you omit the keyword in and the list of arguments, select uses the positional parameters in place of the arg items. The menu is formatted with numbers before each item. For example, a select structure that begins with

select fruit in apple banana blueberry kiwi orange watermelon STOP

displays the following menu:

1) apple       3) blueberry   5) orange      7) STOP
2) banana      4) kiwi        6) watermelon

The select structure uses the values of the LINES and COLUMNS variables to determine the size of the display. (LINES has a default value of 24; COLUMNS has a default value of 80.) With COLUMNS set to 20, the menu looks like this:

1) apple
2) banana
3) blueberry
4) kiwi
5) orange
6) watermelon
7) STOP

PS3

After displaying the menu select displays the value of PS3, the special select prompt. The default value of PS3 is ?# but you typically set PS3 to a more meaningful value. When you enter a valid number (one in the menu range) in response to the PS3 prompt, select sets varname to the argument corresponding to the number you entered. If you make an invalid entry, varname is set to null. Either way select stores your response in the keyword variable REPLY and then executes the commands between do and done. If you press RETURN without entering a choice, the shell redisplays the menu and the PS3 prompt.

The select structure continues to issue the PS3 prompt and execute the commands until something causes it to exittypically a break or exit statement. A break statement exits from the loop and an exit statement exits from the script.

The following script illustrates the use of select:

$ cat fruit2
#!/bin/bash
PS3="Choose your favorite fruit from these possibilities: "
select FRUIT in apple banana blueberry kiwi orange watermelon STOP
do
    if [ "$FRUIT" == "" ]; then
        echo -e "Invalid entry.\n"
        continue
    elif [ $FRUIT = STOP ]; then
        echo "Thanks for playing!"
        break
    fi
echo "You chose $FRUIT as your favorite."
echo -e "That is choice number $REPLY.\n"
done
$ fruit2
1) apple       3) blueberry    5) orange      7) STOP
2) banana      4) kiwi         6) watermelon
Choose your favorite fruit from these possibilities: 3
You chose blueberry as your favorite.
That is choice number 3.

Choose your favorite fruit from these possibilities: 99
Invalid entry.

Choose your favorite fruit from these possibilities: 7
Thanks for playing!

After setting the PS3 prompt and establishing the menu with the select statement, fruit2 executes the commands between do and done. If the user makes an invalid entry, the shell sets varname ($FRUIT) to a null value, so fruit2 first tests whether $FRUIT is null. If it is, echo displays an error and continue causes the shell to redisplay the PS3 prompt. If the entry is valid, the script tests whether the user wants to stop. If so, echo displays a message and break exits from the select structure (and from the script). If the user entered a valid response and does not want to stop, the script displays the name and number of the user's response. (See page 548 for information about the e option to echo.)

Here Document

A Here document allows you to redirect input to a shell script from within the shell script itself. A Here document is so called because it is hereimmediately accessible in the shell scriptinstead of there, perhaps in another file.

The following script, named birthday, contains a Here document. The two less than (<<) symbols in the first line indicate that a Here document follows. One or more characters that delimit the Here document follow the less than symbolsthis example uses a plus sign. Whereas the opening delimiter must appear adjacent to the less than symbols, the closing delimiter must be on a line by itself. The shell sends everything between the two delimiters to the process as standard input. In the example it is as though you had redirected standard input to grep from a file, except that the file is embedded in the shell script:

$ cat birthday
grep -i "$1" <<+
Alex    June 22
Barbara February 3
Darlene May 8
Helen   March 13
Jenny   January 23
Nancy   June 26
+
$ birthday Jenny
Jenny   January 23
$ birthday june
Alex    June 22
Nancy   June 26

When you run birthday, it lists all the Here document lines that contain the argument you called it with. In this case the first time birthday is run, it displays Jenny's birthday because it is called with an argument of Jenny. The second run displays all the birthdays in June. The i argument causes grep's search not to be case sensitive.

Optional

The next script, named bundle,[1] includes a clever use of a Here document. The bundle script is an elegant example of a script that creates a shell archive (shar) file. The script, shown following, creates a file that is itself a shell script containing several other files as well as the code to re-create the original files.

$ cat bundle
#!/bin/bash
# bundle: group files into distribution package

echo "# To unbundle, bash this file"
for i
do
    echo "echo $i 1>&2"
    echo "cat >$i <<'End of $i'"
    cat $i
    echo "End of $i"
done

Just as the shell does not treat special characters that occur in standard input of a shell script as special, so the shell does not treat the special characters that occur between the delimiters in a Here document as special.

As the following example shows, the output of bundle is a shell script, which is redirected to a file named bothfiles. It contains the contents of each file given as an argument to bundle (file1 and file2 in this case) inside a Here document. To extract the original files from bothfiles, you simply run it as an argument to a bash command. Before each Here document is a cat command that causes the Here document to be written to a new file when bothfiles is run:

$ cat file1
This is a file.
It contains two lines.
$ cat file2
This is another file.
It contains
three lines.

$ bundle file1 file2 > bothfiles
$ cat bothfiles
# To unbundle, bash this file
echo file1 1>&2
cat >file1 <<'End of file1'
This is a file.
It contains two lines.
End of file1
echo file2 1>&2
cat >file2 <<'End of file2'
This is another file.
It contains
three lines.
End of file2

In the next example, file1 and file2 are removed before bothfiles is run. The bothfiles script echoes the names of the files it creates as it creates them. The ls command then shows that bothfiles has re-created file1 and file2:

$ rm file1 file2
$ bash bothfiles
file1
file2
$ ls
bothfiles
file1
file2

With its output redirected, the cat utility supports only data forks. Files that have resource forks or other metadata will not be completely backed up by this script. For more information see the "Redirection does not support resource forks" tip on page 94.

[1] Thanks to Brian W. Kernighan and Rob Pike, The Unix Programming Environment (Englewood Cliffs, N.J.: Prentice-Hall, 1984), 98. Reprinted with permission.


File Descriptors

As discussed on page 262, before a process can read from or write to a file it must open that file. When a process opens a file, Mac OS X associates a number (called a file descriptor) with the file. Each process has its own set of open files and its own file descriptors. After opening a file, a process reads from and writes to that file by referring to its file descriptor. When it no longer needs the file, the process closes the file, freeing the file descriptor.

A typical Mac OS X process starts with three open files: standard input (file descriptor 0), standard output (file descriptor 1), and standard error (file descriptor 2). Often those are the only files the process needs. Recall that you redirect standard output with the symbol > or the symbol 1> and that you redirect standard error with the symbol 2>. Although you can redirect other file descriptors, because file descriptors other than 0, 1, and 2 do not have any special conventional meaning, it is rarely useful to do so. The exception is in programs that you write yourself, in which case you control the meaning of the file descriptors and can take advantage of redirection.

Opening a file descriptor

The Bourne Again Shell opens files using the exec builtin as follows:

exec n> outfile
exec m< infile

The first line opens outfile for output and holds it open, associating it with file descriptor n. The second line opens infile for input and holds it open, associating it with file descriptor m.

Duplicating a file descriptor

The <& token duplicates an input file descriptor; use >& to duplicate an output file descriptor. You can duplicate a file descriptor by making it refer to the same file as another open file descriptor, such as standard input or output. Use the following format to open or redirect file descriptor n as a duplicate of file descriptor m:

exec n<&m

Once you have opened a file, you can use it for input and output in two different ways. First, you can use I/O redirection on any command line, redirecting standard output to a file descriptor with >&n or redirecting standard input from a file descriptor with <&n. Second, you can use the read (page 571) and echo builtins. If you invoke other commands, including functions (page 314), they inherit these open files and file descriptors. When you have finished using a file, you can close it with

exec n<&

When you invoke the shell function in the next example, named mycp, with two arguments, it copies the file named by the first argument to the file named by the second argument. If you supply only one argument, the script copies the file named by the argument to standard output. If you invoke mycp with no arguments, it copies standard input to standard output.

Tip: A function is not a shell script

The mycp example is a shell function; it will not work as you expect if you execute it as a shell script. (It will work: The function will be created in a very short-lived subshell, which is probably of little use.) You can enter this function from the keyboard. If you put the function in a file, you can run it as an argument to the . (dot) builtin (page 261). You can also put the function in a startup file if you want it to be always available (page 315).

function mycp ()
{
case $# in
    0)
        # zero arguments
        # file descriptor 3 duplicates standard input
        # file descriptor 4 duplicates standard output
        exec 3<&0 4<&1
        ;;
    1)
        # one argument
        # open the file named by the argument for input
        # and associate it with file descriptor 3
        # file descriptor 4 duplicates standard output
        exec 3< $1 4<&1
        ;;
    2)
        # two arguments
        # open the file named by the first argument for input
        # and associate it with file descriptor 3
        # open the file named by the second argument for output
        # and associate it with file descriptor 4
        exec 3< $1 4> $2
        ;;
    *)
        echo "Usage: mycp [source [dest]]"
        return 1
        ;;
esac

# call cat with input coming from file descriptor 3
# and output going to file descriptor 4
cat <&3 >&4

# close file descriptors 3 and 4
exec 3<&- 4<&-
}

The real work of this function is done in the line that begins with cat. The rest of the script arranges for file descriptors 3 and 4, which are the input and output of the cat command, to be associated with the appropriate files.

With its output redirected, the cat utility supports only data forks. Files that have resource forks or other metadata will not be copied properly by this script. For more information see the "Redirection does not support resource forks" tip on page 94.

Optional

The next program takes two filenames on the command line, sorts both, and sends the output to temporary files. The program then merges the sorted files to standard output, preceding each line by a number that indicates which file it came from.

$ cat sortmerg
#!/bin/bash
usage ()
{
if [ $# -ne 2 ]; then
    echo "Usage: $0 file1 file2" 2>&1
    exit 1
    fi
}

# Default temporary directory
: ${TEMPDIR:=/tmp}

# Check argument count
usage "$@"

# Set up temporary files for sorting
file1=$TEMPDIR/$$.file1
file2=$TEMPDIR/$$.file2

# Sort
sort $1 > $file1
sort $2 > $file2

# Open $file1 and $file2 for reading. Use file descriptors 3 and 4.
exec 3<$file1
exec 4<$file2

# Read the first line from each file to figure out how to start.
read Line1 <&3
status1=$?
read Line2 <&4
status2=$?

# Strategy: while there is still input left in both files:
#   Output the line that should come first.
#   Read a new line from the file that line came from.
while [ $status1 -eq 0 -a $status2 -eq 0 ]
    do
       if [[ "$Line2" > "$Line1" ]]; then
           echo -e "1.\t$Line1"
           read -u3 Line1
           status1=$?
        else
           echo -e "2.\t$Line2"
           read -u4 Line2
           status2=$?
        fi
    done

# Now one of the files is at end-of-file.
# Read from each file until the end.
# First file1:
while [ $status1 -eq 0 ]
    do
        echo -e "1.\t$Line1"
        read Line1 <&3
        status1=$?
    done
# Next file2:
while [[ $status2 -eq 0 ]]
    do
        echo -e "2.\t$Line2"
        read Line2 <&4
        status2=$?
    done

# Close and remove both input files
exec 3<&- 4<&-
rm -f $file1 $file2
exit 0

Parameters and Variables

Shell parameters and variables were introduced on page 278. This section adds to the previous coverage with a discussion of array variables, global versus local variables, special and positional parameters, and expanding null and unset variables.

Array Variables

The Bourne Again Shell supports one-dimensional array variables. The subscripts are integers with zero-based indexing (i.e., the first element of the array has the subscript 0). The following format declares and assigns values to an array:

name=(element1 element2 ...)

The following example assigns four values to the array NAMES:

$ NAMES=(max helen sam zach)

You reference a single element of an array as follows:

$ echo ${NAMES[2]}
sam

The subscripts [*] and [@] both extract the entire array but work differently when used within double quotation marks. An @ produces an array that is a duplicate of the original array; an * produces a single element of an array (or a plain variable) that holds all the elements of the array separated by the first character in IFS (normally a SPACE). In the following example, the array A is filled with the elements of the NAMES variable using an *, and B is filled using an @. The declare builtin with the a option displays the values of the arrays (and reminds you that bash uses zero-based indexing for arrays):

$ A=("${NAMES[*]}")
$ B=("${NAMES[@]}")

$ declare -a
declare -a A='([0]="max helen sam zach")'
declare -a B='([0]="max" [1]="helen" [2]="sam" [3]="zach")'
...
declare -a NAMES='([0]="max" [1]="helen" [2]="sam" [3]="zach")'

From the output of declare, you can see that NAMES and B have multiple elements. In contrast, A, which was assigned its value with an * within double quotation marks, has only one element: A has all its elements enclosed between double quotation marks.

In the next example, echo attempts to display element 1 of array A. Nothing is displayed because A has only one element and that element has an index of 0. Element 0 of array A holds all four names. Element 1 of B holds the second item in the array and element 0 holds the first item.

$ echo ${A[1]}

$ echo ${A[0]}
max helen sam zach
$ echo ${B[1]}
helen
$ echo ${B[0]}
max

You can apply the ${#name[*]} operator to array variables, returning the number of elements in the array:

$ echo ${#NAMES[*]}
4

The same operator, when given the index of an element of an array in place of *, returns the length of the element:

$ echo ${#NAMES[1]}
5

You can use subscripts on the left side of an assignment statement to replace selected elements of the array:

$ NAMES[1]=alex
$ echo ${NAMES[*]}
max alex sam zach

Locality of Variables

By default variables are local to the process in which they are declared. Thus a shell script does not have access to variables declared in your login shell unless you explicitly make the variables available (global). Under bash, export makes a variable available to child processes. Under tcsh, setenv (page 353) assigns a value to a variable and makes it available to child processes. The examples in this section use the bash syntax but the theory applies to both shells.

Once you use the export builtin with a variable name as an argument, the shell places the value of the variable in the calling environment of child processes. This call by value gives each child process a copy of the variable for its own use.

The following extest1 shell script assigns a value of american to the variable named cheese and then displays its filename (extest1) and the value of cheese. The extest1 script then calls subtest, which attempts to display the same information. Next subtest declares a cheese variable and displays its value. When subtest finishes, it returns control to the parent process, which is executing extest1. At this point extest1 again displays the value of the original cheese variable.

$ cat extest1
cheese=american
echo "extest1 1: $cheese"
subtest
echo "extest1 2: $cheese"
$ cat subtest
echo "subtest 1: $cheese"
cheese=swiss
echo "subtest 2: $cheese"
$ extest1
extest1 1: american
subtest 1:
subtest 2: swiss
extest1 2: american

The subtest script never receives the value of cheese from extest1, and extest1 never loses the value. Unlike in the real world, a child can never affect its parent's attributes. When a process attempts to display the value of a variable that has not been declared, as is the case with subtest, the process displays nothing; the value of an undeclared variable is that of a null string.

The following extest2 script is the same as extest1 except that it uses export to make cheese available to the subtest script:

$ cat extest2
export cheese=american
echo "extest2 1: $cheese"
subtest
echo "extest2 2: $cheese"
$ extest2
extest2 1: american
subtest 1: american
subtest 2: swiss
extest2 2: american

Here the child process inherits the value of cheese as american and, after displaying this value, changes its copy to swiss. When control is returned to the parent, the parent's copy of cheese retains its original value: american.

An export builtin can optionally include an assignment:

export cheese=american

The preceding statement is equivalent to the following two statements:

cheese=american
export cheese

Although it is rarely done, you can export a variable before you assign a value to it. You do not need to export an already-exported variable a second time after you change its value. For example, you do not usually need to export PATH when you assign a value to it in ~/.bash_profile because it is typically exported in the /etc/profile global startup file.

Functions

Because functions run in the same environment as the shell that calls them, variables are implicitly shared by a shell and a function it calls.

$ function nam () {
> echo $myname
> myname=zach
> }

$ myname=sam
$ nam
sam
$ echo $myname
zach

In the preceding example, the myname variable is set to sam in the interactive shell. Then the nam function is called. It displays the value of myname it has (sam) and sets myname to zach. The final echo shows that, in the interactive shell, the value of myname has been changed to zach.

Function local variables

Local variables are helpful in a function written for general use. Because the function is called by many scripts that may be written by different programmers, you need to make sure that the names of the variables used within the function do not interact with variables of the same name in the programs that call the function. Local variables eliminate this problem. When used within a function, the typeset builtin declares a variable to be local to the function it is defined in.

The next example shows the use of a local variable in a function. It uses two variables named count. The first is declared and assigned a value of 10 in the interactive shell. Its value never changes, as echo verifies after count_down is run. The other count is declared, using typeset, to be local to the function. Its value, which is unknown outside the function, ranges from 4 to 1, as the echo command within the function confirms.

The example shows the function being entered from the keyboard; it is not a shell script. (See the tip "A function is not a shell script" on page 556).

$ function count_down () {
> typeset count
> count=$1
> while [ $count -gt 0 ]
> do
> echo "$count..."
> ((count=count-1))
> sleep 1
> done
> echo "Blast Off."
> }
$ count=10
$ count_down 4
4...
3...
2...
1...
Blast Off\!
$ echo $count
10

The ((count=count1)) assignment is enclosed between double parentheses, which cause the shell to perform an arithmetic evaluation (page 585). Within the double parentheses you can reference shell variables without the leading dollar sign ($).

Special Parameters

Special parameters enable you to access useful values pertaining to command line arguments and the execution of shell commands. You reference a shell special parameter by preceding a special character with a dollar sign ($). As with positional parameters, it is not possible to modify the value of a special parameter by assignment.

$$: PID Number

The shell stores in the $$ parameter the PID number of the process that is executing it. In the following interaction, echo displays the value of this variable and the ps utility confirms its value. Both commands show that the shell has a PID number of 5209:

$ echo $$
5209
$ ps -a
  PID TT STAT TIME      COMMAND
 1709 p1 Ss     0:00.32 -bash
 4168 p1 R+     0:00.00 ps -a

Because echo is built into the shell, the shell does not have to create another process when you give an echo command. However, the results are the same whether echo is a builtin or not, because the shell substitutes the value of $$ before it forks a new process to run a command. Try using the echo utility (/bin/echo), which is run by another process, and see what happens. In the following example, the shell substitutes the value of $$ and passes that value to cp as a prefix for a filename:

$ echo $$
8232
$ cp memo $$.memo
$ ls
8232.memo memo

Incorporating a PID number in a filename is useful for creating unique filenames when the meanings of the names do not matter; it is often used in shell scripts for creating names of temporary files. When two people are running the same shell script, these unique filenames keep them from inadvertently sharing the same temporary file.

The following example demonstrates that the shell creates a new shell process when it runs a shell script. The id2 script displays the PID number of the process running it (not the process that called itthe substitution for $$ is performed by the shell that is forked to run id2):

$ cat id2
echo "$0 PID= $$"
$ echo $$
8232
$ id2
./id2 PID= 8362
$ echo $$
8232

The first echo displays the PID number of the interactive shell. Then id2 displays its name ($0) and the PID of the subshell that it is running in. The last echo shows that the PID number of the interactive shell has not changed.

$!

The value of the PID number of the last process that you ran in the background is stored in $! (not available in tcsh). The following example executes sleep as a background task and uses echo to display the value of $!:

$ sleep 60 &
[1] 8376
$ echo $!
8376

$?: Exit Status

When a process stops executing for any reason, it returns an exit status to the parent process. The exit status is also referred to as a condition code or a return code. The $? ($status under tcsh) variable stores the exit status of the last command.

By convention a nonzero exit status represents a false value and means that the command failed. A zero is true and indicates that the command was successful. In the following example, the first ls command succeeds and the second fails:

$ ls es
es
$ echo $?
0
$ ls xxx
ls: xxx: No such file or directory
$ echo $?
1

You can specify the exit status that a shell script returns by using the exit builtin, followed by a number, to terminate the script. If you do not use exit with a number to terminate a script, the exit status of the script is that of the last command the script ran.

$ cat es
echo This program returns an exit status of 7.
exit 7
$ es
This program returns an exit status of 7.
$ echo $?
7
$ echo $?
0

The es shell script displays a message and terminates execution with an exit command that returns an exit status of 7, the user-defined exit status in this script. The first echo then displays the value of the exit status of es. The second echo displays the value of the exit status of the first echo. The value is 0 because the first echo was successful.

Positional Parameters

The positional parameters comprise the command name and command line arguments. They are called positional because within a shell script, you refer to them by their position on the command line. Only the set builtin (page 568) allows you to change the values of positional parameters with one exception: You cannot change the value of the command name from within a script. The tcsh set builtin does not change the values of positional parameters.

$#: Number of Command Line Arguments

The $# parameter holds the number of arguments on the command line (positional parameters), not counting the command itself:

$ cat num_args
echo "This script was called with $# arguments."
$ num_args sam max zach
This script was called with 3 arguments.

$0: Name of the Calling Program

The shell stores the name of the command you used to call a program in parameter $0. This parameter is numbered zero because it appears before the first argument on the command line:

$ cat abc
echo "The command used to run this script is $0"
$ abc
The command used to run this script is ./abc
$ /Users/sam/abc
The command used to run this script is /Users/sam/abc

The preceding shell script uses echo to verify the name of the script you are executing. You can use the basename utility and command substitution to extract and display the simple filename of the command:

$ cat abc2
echo "The command used to run this script is $(basename $0)"
$ /Users/sam/abc2
The command used to run this script is abc2

$1-$n: Command Line Arguments

The first argument on the command line is represented by parameter $1, the second argument by $2, and so on up to $n. For values of n over 9, the number must be enclosed within braces. For example, the twelfth command line argument is represented by ${12}. The following script displays positional parameters that hold command line arguments:

$ cat display_5args
echo First 5 arguments are $1 $2 $3 $4 $5

$ display_5args jenny alex helen
First 5 arguments are jenny alex helen

The display_5args script displays the first five command line arguments. The shell assigns a null value to each parameter that represents an argument that is not present on the command line. Thus the $4 and $5 variables have null values in this example.

$*

The $* variable represents all the command line arguments, as the display_all program demonstrates:

$ cat display_all
echo All arguments are $*

$ display_all a b c d e f g h i j k l m n o p
All arguments are a b c d e f g h i j k l m n o p

Enclose references to positional parameters between double quotation marks. The quotation marks are particularly important when you are using positional parameters as arguments to commands. Without double quotation marks, a positional parameter that is not set or that has a null value disappears:

$ cat showargs
echo "$0 was called with $# arguments, the first is :$1:."

$ showargs a b c
./showargs was called with 3 arguments, the first is :a:.
$ echo $xx

$ showargs $xx a b c
./showargs was called with 3 arguments, the first is :a:.
$ showargs "$xx" a b c
./showargs was called with 4 arguments, the first is ::.

The showargs script displays the number of arguments ($#) followed by the value of the first argument enclosed between colons. The preceding example first calls showargs with three simple arguments. Next the echo command demonstrates that the $xx variable, which is not set, has a null value. In the final two calls to showargs, the first argument is $xx. In the first case the command line becomes showargs a b c; the shell passes showargs three arguments. In the second case the command line becomes showargs "" a b c, which results in calling showargs with four arguments. The difference in the two calls to showargs illustrates a subtle potential problem that you should keep in mind when using positional parameters that may not be set or that may have a null value.

"$*"versus "$@"

The $* and $@ parameters work the same way except when they are enclosed within double quotation marks. Using "$*" yields a single argument (with SPACEs or the value of IFS [page 288] between the positional parameters), whereas "$@" produces a list wherein each positional parameter is a separate argument. This difference typically makes "$@" more useful than "$*" in shell scripts.

The following scripts help to explain the difference between these two special parameters. In the second line of both scripts, the single quotation marks keep the shell from interpreting the enclosed special characters so they can be displayed as themselves. The bb1 script shows that set "$*" assigns multiple arguments to the first command line parameter:

$ cat bb1
set "$*"
echo $# parameters with '"$*"'
echo 1: $1
echo 2: $2
echo 3: $3

$ bb1 a b c
1 parameters with "$*"'
1: a b c
2:
3:

The bb2 script shows that set "$@" assigns each argument to a different command line parameter:

$ cat bb2
set "$@"
echo $# parameters with '"$@"'
echo 1: $1
echo 2: $2
echo 3: $3

$ bb2 a b c
3 parameters with "$@"
1: a
2: b
3: c

shift: Promotes Command Line Arguments

The shift builtin promotes each command line argument. The first argument (which was $1) is discarded. The second argument (which was $2) becomes the first argument (now $1), the third becomes the second, and so on. Because no "unshift" command exists, you cannot bring back arguments that have been discarded. An optional argument to shift specifies the number of positions to shift (and the number of arguments to discard); the default is 1.

The following demo_shift script is called with three arguments. Double quotation marks around the arguments to echo preserve the spacing of the output. The program displays the arguments and shifts them repeatedly until there are no more arguments left to shift:

$ cat demo_shift
echo "arg1= $1    arg2= $2    arg3= $3"
shift
echo "arg1= $1    arg2= $2    arg3= $3"
shift
echo "arg1= $1    arg2= $2    arg3= $3"
shift
echo "arg1= $1    arg2= $2    arg3= $3"
shift

$ demo_shift alice helen jenny
arg1= alice    arg2= helen    arg3= jenny
arg1= helen    arg2= jenny    arg3=
arg1= jenny    arg2=    arg3=
arg1=     arg2=    arg3=

Repeatedly using shift is a convenient way to loop over all the command line arguments in shell scripts that expect an arbitrary number of arguments. See page 529 for a shell script that uses shift.

set: Initializes Command Line Arguments

When you call the set builtin with one or more arguments, it assigns the values of the arguments to the positional parameters, starting with $1 (not available in tcsh). The following script uses set to assign values to the positional parameters $1, $2, and $3:

$ cat set_it
set this is it
echo $3 $2 $1
$ set_it
it is this

Combining command substitution (page 327) with the set builtin is a convenient way to get standard output of a command in a form that can be easily manipulated in a shell script. The following script shows how to use date and set to provide the date in a useful format. The first command shows the output of date. Then cat displays the contents of the dateset script. The first command in this script uses command substitution to set the positional parameters to the output of the date utility. The next command, echo $*, displays all positional parameters resulting from the previous set. Subsequent commands display the values of parameters $1, $2, $3, and $4. The final command displays the date in a format you can use in a letter or report:

$ date
Wed Jan  5 23:39:18 PST 2005
$ cat dateset
set $(date)
echo $*
echo
echo "Argument 1: $1"
echo "Argument 2: $2"
echo "Argument 3: $3"
echo "Argument 6: $6"
echo
echo "$2 $3, $6"

$ dateset
Wed Jan 5 23:39:25 PST 2005

Argument 1: Wed
Argument 2: Jan
Argument 3: 5
Argument 6: 2005

Jan 5, 2005

You can also use the +format argument to date (page 701) to modify the format of its output.

When used without any arguments, set displays a list of the shell variables that are set, including user-created variables and keyword variables. Under bash, this list is the same as that displayed by declare and typeset when they are called without any arguments.

The set builtin also accepts options that let you customize the behavior of the shell (not available in tcsh). For more information refer to "set ±o: Turns Shell Features On and Off" on page 318.

Expanding Null and Unset Variables

The expression ${name} (or just $name if it is not ambiguous) expands to the value of the name variable. If name is null or not set, bash expands ${name} to a null string. The Bourne Again Shell provides the following alternatives to accepting the expanded null string as the value of the variable:

  • Use a default value for the variable.

  • Use a default value and assign that value to the variable.

  • Display an error.

You can choose one of these alternatives by using a modifier with the variable name. In addition, you can use set o nounset (page 320) to cause bash to display an error and exit from a script whenever an unset variable is referenced.

:- Uses a Default Value

The :- modifier uses a default value in place of a null or unset variable while allowing a nonnull variable to represent itself:

${name:default}

The shell interprets : as "If name is null or unset, expand default and use the expanded value in place of name; else use name." The following command lists the contents of the directory named by the LIT variable. If LIT is null or unset, it lists the contents of /Users/alex/literature:

$ ls ${LIT:-/Users/alex/literature}

The default can itself have variable references that are expanded:

$ ls ${LIT:-$HOME/literature}

:= Assigns a Default Value

The : modifier does not change the value of a variable. You may want to change the value of a null or unset variable to its default in a script, however. You can do so with the := modifier:

${name:=default}

The shell expands the expression ${name:=default} in the same manner as it expands ${name:default} but also sets the value of name to the expanded value of default. If a script contains a line such as the following and LIT is unset or null at the time this line is executed, LIT is assigned the value /Users/alex/literature:

$ ls ${LIT:=/Users/alex/literature}

: builtin

Shell scripts frequently start with the : (colon) builtin followed on the same line by the := expansion modifier to set any variables that may be null or unset. The : builtin evaluates each token in the remainder of the command line but does not execute any commands. Without the leading colon (:), the shell evaluates and attempts to execute the "command" that results from the evaluation.

Use the following syntax to set a default for a null or unset variable in a shell script (there is a SPACE following the first colon):

: ${name:=default}

When a script needs a directory for temporary files and uses the value of TEMPDIR for the name of this directory, the following line makes TEMPDIR default to /tmp:

: ${TEMPDIR:=/tmp}

:? Displays an Error Message

Sometimes a script needs the value of a variable but you cannot supply a reasonable default at the time you write the script. If the variable is null or unset, the :? modifier causes the script to display an error message and terminate with an exit status of 1:

${name:?message}

You must quote message if it contains SPACEs. If you omit message, the shell displays the default error message (parameter null or not set). Interactive shells do not exit when you use :?. In the following command, TESTDIR is not set so the shell displays on standard error the expanded value of the string following :?. In this case the string includes command substitution for date, with the %T format being followed by the string error, variable not set.

cd ${TESTDIR:?$(date +%T) error, variable not set.}
bash: TESTDIR: 16:16:14 error, variable not set.

Builtin Commands

Builtin commands were introduced in Chapter 5. Commands that are built into a shell do not fork a new process when you execute them. This section discusses the type, read, exec, trap, kill, and getopts builtins and concludes with Table 13-6 on page 583, which lists many bash builtins. See Table 9-10 on page 373 for a list of tcsh builtins.

type: Displays Information About a Command

The type builtin (use which under tcsh) provides information about a command:

$ type cat echo who if lt
cat is hashed (/bin/cat)
echo is a shell builtin
who is /usr/bin/who
if is a shell keyword
lt is aliased to 'ls -ltrh tail'

The preceding output shows the files that would be executed if you gave cat or who as a command. Because cat has already been called from the current shell, it is in the hash table (page 935) and type reports that cat is hashed. The output also shows that a call to echo runs the echo builtin, if is a keyword, and lt is an alias.

read: Accepts User Input

When you begin writing shell scripts, you soon realize that one of the most common tasks for user-created variables is storing information a user enters in response to a prompt. Using read, scripts can accept input from the user and store that input in variables. See page 358 for information about reading user input under tcsh. The read builtin reads one line from standard input and assigns the words on the line to one or more variables:

$ cat read1
echo -n "Go ahead: "
read firstline
echo "You entered: $firstline"
$ read1
Go ahead: This is a line.
You entered: This is a line.

The first line of the read1 script uses echo to prompt you to enter a line of text. The n option suppresses the following NEWLINE, allowing you to enter a line of text on the same line as the prompt. The second line reads the text into the variable firstline. The third line verifies the action of read by displaying the value of firstline. The variable is quoted (along with the text string) in this example because you, as the script writer, cannot anticipate which characters the user might enter in response to the prompt. Consider what would happen if the variable were not quoted and the user entered * in response to the prompt:

$ cat read1_no_quote
echo -n "Go ahead: "
read firstline
echo You entered: $firstline
$ read1_no_quote
Go ahead: *
You entered: read1 read1_no_quote script.1
$ ls
read1   read1_no_quote    script.1

The ls command lists the same words as the script, demonstrating that the shell expands the asterisk into a list of files in the working directory. When the variable $firstline is surrounded by double quotation marks, the shell does not expand the asterisk. Thus the read1 script behaves correctly:

$ read1
Go ahead: *
You entered: *

If you want the shell to interpret the special meanings of special characters, do not use quotation marks.

REPLY

The read builtin has features that can make it easier to use. When you do not specify a variable to receive read's input, bash puts the input into the variable named REPLY. You can use the p option to prompt the user instead of using a separate echo command. The following read1a script performs exactly the same task as read1:

$ cat read1a
read -p "Go ahead: "
echo "You entered: $REPLY"

The read2 script prompts for a command line and reads the user's response into the variable cmd. The script then attempts to execute the command line that results from the expansion of the cmd variable:

$ cat read2
read -p "Enter a command: " cmd
$cmd
echo "Thanks"

In the following example, read2 reads a command line that calls the echo builtin. The shell executes the command and then displays Thanks. Next read2 reads a command line that executes the who utility:

$ read2
Enter a command: echo Please display this message.
Please display this message.
Thanks
$ read2
Enter a command: who
alex     console  Jun 10 15:20
alex     ttyp1    Jun 16 15:16 (bravo.example.com)
Thanks

If cmd does not expand into a valid command line, the shell issues an error message:

$ read2
Enter a command: xxx
./read2: line 2: xxx: command not found
Thanks

The read3 script reads values into three variables. The read builtin assigns one word (a sequence of nonblank characters) to each variable:

$ cat read3
read -p "Enter something: " word1 word2 word3
echo "Word 1 is: $word1"
echo "Word 2 is: $word2"
echo "Word 3 is: $word3"

$ read3
Enter something: this is something
Word 1 is: this
Word 2 is: is
Word 3 is: something

When you enter more words than read has variables, read assigns one word to each variable, with all leftover words going to the last variable. Both read1 and read2 assigned the first word and all leftover words to the one variable they each had to work with. In the following example, read accepts five words into three variables, assigning the first word to the first variable, the second word to the second variable, and the third through fifth words to the third variable:

$ read3
Enter something: this is something else, really.
Word 1 is:  this
Word 2 is:  is
Word 3 is:  something else, really.

Table 13-4 lists some of the options supported by the read builtin.

Table 13-4. read options

Option

Function

a aname (array)

Assigns each word of input to an element of array aname.

d delim (delimiter)

Uses delim to terminate the input instead of NEWLINE.

e (Readline)

If input is coming from a keyboard, use the Readline Library (page 304) to get input.

n num (number of characters)

Reads num characters and returns. As soon as the user types num characters, read returns; there is no need to press RETURN.

p prompt (prompt)

Displays prompt on standard error without a terminating NEWLINE before reading input. Displays prompt only when input comes from the keyboard.

s (silent)

Does not echo characters.

un (file descriptor)

Uses the integer n as the file descriptor that read takes its input from.

    read u4 arg1 arg2

is equivalent to

    read arg1 arg2 <&4

See "File Descriptors" (page 555) for a discussion of redirection and file descriptors.

The read builtin returns an exit status of 0 if it successfully reads any data. It has a nonzero exit status when it reaches the EOF (end of file). The following example runs a while loop from the command line. It takes its input from the names file and terminates after reading the last line from names.

$ cat names
Alice Jones
Robert Smith
Alice Paulson
John Q. Public
$ while read first rest
> do
> echo $rest, $first
> done < names
Jones, Alice
Smith, Robert
Paulson, Alice
Q. Public, John
$

The placement of the redirection symbol (<) for the while structure is critical. It is important that you place the redirection symbol at the done statement and not at the call to read.

Optional

Each time you redirect input, the shell opens the input file and repositions the read pointer at the start of the file:

$ read line1 < names; echo $line1; read line2 < names; echo $line2
Alice Jones
Alice Jones

Here each read opens names and starts at the beginning of the names file. In the following example, names is opened once, as standard input of the subshell created by the parentheses. Each read then reads successive lines of standard input.

$ (read line1; echo $line1; read line2; echo $line2) < names
Alice Jones
Robert Smith

Another way to get the same effect is to open the input file with exec and hold it open (refer to "File Descriptors" on page 555):

$ exec 3< names
$ read -u3 line1; echo $line1; read -u3 line2; echo $line2
Alice Jones
Robert Smith
$ exec
3<&-

exec: Executes a Command

The exec builtin (not available in tcsh) has two primary purposes: to run a command without creating a new process and to redirect a file descriptorincluding standard input, output, or errorof a shell script from within the script (page 555). When the shell executes a command that is not built into the shell, it typically creates a new process. The new process inherits environment (global or exported) variables from its parent but does not inherit variables that are not exported by the parent. (For more information refer to "Locality of Variables" on page 560.) In contrast, exec executes a command in place of (overlays) the current process.

exec versus .(dot)

Insofar as exec runs a command in the environment of the original process, it is similar to the .(dot) command (page 261). However, unlike the . command, which can run only shell scripts, exec can run both scripts and compiled programs. Also, whereas the . command returns control to the original script when it finishes running, exec does not. Finally, the . command gives the new program access to local variables, whereas exec does not.

exec runs a command

The exec builtin used for running a command has the following syntax:

exec command arguments

exec does not return control

Because the shell does not create a new process when you use exec, the command runs more quickly. However, because exec does not return control to the original program, it can be used only as the last command that you want to run in a script. The following script shows that control is not returned to the script:

$ cat exec_demo
who
exec date
echo "This line is never displayed."

$ exec_demo
jenny    ttyp1    May 30 7:05 (bravo.example.com)
hls      ttyp2    May 30 6:59
Mon May 30 11:42:56 PDT 2005

The next example, a modified version of the out script (page 529), uses exec to execute the final command the script runs. Because out runs either cat or less and then terminates, the new version, named out2, uses exec with both cat and less:

$ cat out2
if [ $# -eq 0 ]
    then
        echo "Usage: out2 [-v] filenames" 1>&2
        exit 1
fi
if [ "$1" = "-v" ]
    then
        shift
        exec less "$@"
    else
        exec cat -- "$@"

fi

exec redirects input and output

The second major use of exec is to redirect a file descriptorincluding standard input, output, or errorfrom within a script. The next command causes all subsequent input to a script that would have come from standard input to come from the file named infile:

exec < infile

Similarly the following command redirects standard output and standard error to outfile and errfile, respectively:

exec > outfile 2> errfile

When you use exec in this manner, the current process is not replaced with a new process, and exec can be followed by other commands in the script.

/dev/tty

When you redirect the output from a script to a file, you must make sure that the user sees any prompts the script displays. The /dev/tty device is a pseudonym for the screen the user is working on; you can use this device to refer to the user's screen without knowing which device it is. (The tty utility displays the name of the device you are using.) By redirecting the output from a script to /dev/tty, you ensure that prompts and messages go to the user's terminal, regardless of which terminal the user is logged in on. Messages sent to /dev/tty are also not diverted if standard output and standard error from the script are redirected.

The to_screen1 script sends output to three places: standard output, standard error, and the user's screen. When it is run with standard output and standard error redirected, to_screen1 still displays the message sent to /dev/tty on the user's screen. The out and err files hold the output sent to standard output and standard error.

$ cat to_screen1
echo "message to standard output"
echo "message to standard error" 1>&2
echo "message to the user" > /dev/tty

$ to_screen1 > out 2> err
message to the user
$ cat out
message to standard output
$ cat err
message to standard error

The following command redirects the output from a script to the user's screen:

exec > /dev/tty

Putting this command at the beginning of the previous script changes where the output goes. In to_screen2, exec redirects standard output to the user's screen so the >/dev/tty is superfluous. Following the exec command, all output sent to standard output goes to /dev/tty (the screen). Output to standard error is not affected.

$ cat to_screen2
exec > /dev/tty
echo "message to standard output"
echo "message to standard error" 1>&2
echo "message to the user" > /dev/tty

$ to_screen2 > out 2> err
message to standard output
message to the user

One disadvantage of using exec to redirect the output to /dev/tty is that all subsequent output is redirected unless you use exec again in the script.

You can also redirect the input to read (standard input) so that it comes from /dev/tty (the keyboard):

read name < /dev/tty

or

exec < /dev/tty

trap: Catches a Signal

A signal is a report to a process about a condition. Mac OS X uses signals to report interrupts generated by the user (for example, pressing the interrupt key) as well as bad system calls, broken pipes, illegal instructions, and other conditions. The trap builtin (tcsh uses onintr) catches, or traps, one or more signals, allowing you to direct the actions a script takes when it receives a specified signal.

This discussion covers six signals that are significant when you work with shell scripts. Table 13-5 lists these signals, the signal numbers that systems often ascribe to them, and the conditions that usually generate each signal. Give the command kill l, trap l, or man 7 signal for a list of signal names.

Table 13-5. Signals

Type

Name

Number

Generating condition

Not a real signal

EXIT

0

Exit because of exit command or reaching the end of the program (not an actual signal but useful in trap)

Hang up

SIGHUP or HUP

1

Disconnect the line

Terminal interrupt

SIGINT or INT

2

Press the interrupt key (usually CONTROL-C)

Quit

SIGQUIT or QUIT

3

Press the quit key (usually CONTROL-SHIFT-| or CONTROL-SHIFT-\)

Kill

SIGKILL or KILL

9

The kill command with the 9 option (cannot be trapped; use only as a last resort)

Software termination

SIGTERM or TERM

15

Default of the kill command

Stop

SIGTSTP or TSTP

20

Press the suspend key (usually CONTROL-Z)

Debug

DEBUG

Executes commands specified in the trap statement after each command (not an actual signal but useful in trap)

Error

ERR

Executes commands specified in the trap statement after each command that returns a nonzero exit status (not an actual signal but useful in trap)

When it traps a signal, a script takes whatever action you specify: It can remove files or finish any other processing as needed, display a message, terminate execution immediately, or ignore the signal. If you do not use trap in a script, any of the six actual signals listed in Table 13-5 (not EXIT, DEBUG, or ERR) terminates the script. Because a process cannot trap a KILL signal, you can use kill KILL (or kill 9) as a last resort to terminate a script or any other process. (See page 580 for more information on kill.)

The trap command has the following syntax:

trap ['commands'] [signal]

The optional commands part specifies the commands that the shell executes when it catches one of the signals specified by signal. The signal can be a signal name or numberfor example, INT or 2. If commands is not present, trap resets the trap to its initial condition, which is usually to exit from the script.

The trap builtin does not require single quotation marks around commands as shown in the preceding syntax, but it is a good practice to use them. The single quotation marks cause shell variables within the commands to be expanded when the signal occurs, not when the shell evaluates the arguments to trap. Even if you do not use any shell variables in the commands, you need to enclose any command that takes arguments within either single or double quotation marks. Quoting the commands causes the shell to pass to TRap the entire command as a single argument.

After executing the commands, the shell resumes executing the script where it left off. If you want trap to prevent a script from exiting when it receives a signal but not to run any commands explicitly, you can specify a null (empty) commands string, as shown in the locktty script (page 543). The following command traps signal number 15 after which the script continues.

trap '' 15

The following script demonstrates how the trap builtin can catch the terminal interrupt signal (2). You can use SIGINT, INT, or 2 to specify this signal. The script returns an exit status of 1:

$ cat inter
#!/bin/bash
trap 'echo PROGRAM INTERRUPTED; exit 1' INT
while true
do
    echo "Program running."
    sleep 1
done
$ inter
Program running.
Program running.
Program running.
CONTROL-C
^CPROGRAM INTERRUPTED
$

:(null) builtin

The second line of inter sets up a trap for the terminal interrupt signal using INT. When trap catches the signal, the shell executes the two commands between the single quotation marks in the TRap command. The echo builtin displays the message PROGRAM INTERRUPTED, exit terminates the shell running the script, and the parent shell displays a prompt. If exit were not there, the shell would return control to the while loop after displaying the message. The while loop repeats continuously until the script receives a signal because the true utility always returns a true exit status. In place of true you can use the : (null) builtin, which is written as a colon and always returns a 0 (true) status.

The trap builtin frequently removes temporary files when a script is terminated prematurely so that the files are not left to clutter the filesystem. The following shell script, named addbanner, uses two traps to remove a temporary file when the script terminates normally or owing to a hangup, software interrupt, quit, or software termination signal:

$ cat addbanner
#!/bin/bash
script=$(basename $0)

if [ ! -r "$HOME/banner" ]
    then
        echo "$script: need readable $HOME/banner file" 1>&2
        exit 1
fi

trap 'exit 1' 1 2 3 15
trap 'rm /tmp/$$.$script 2> /dev/null' 0

for file
do
    if [ -r "$file" -a -w "$file" ]
        then
            cat $HOME/banner $file > /tmp/$$.$script
            cp /tmp/$$.$script $file
            echo "$script: banner added to $file" 1>&2
        else
            echo "$script: need read and write permission for $file" 1>&2
        fi
done

When called with one or more filename arguments, addbanner loops through the files, adding a header to the top of each. This script is useful when you use a standard format at the top of your documents, such as a standard layout for memos, or when you want to add a standard header to shell scripts. The header is kept in a file named ~/banner. Because addbanner uses the HOME variable, which contains the pathname of the user's home directory, the script can be used by several users without modification. If Alex had written the script with /Users/alex in place of $HOME and then given the script to Jenny, either she would have had to change it or addbanner would have used Alex's banner file when Jenny ran it (assuming Jenny had read permission for the file).

The first trap in addbanner causes it to exit with a status of 1 when it receives a hangup, software interrupt (terminal interrupt or quit signal), or software termination signal. The second trap uses a 0 in place of signal-number, which causes trap to execute its command argument whenever the script exits because it receives an exit command or reaches its end. Together these TRaps remove a temporary file whether the script terminates normally or prematurely. Standard error of the second trap is sent to /dev/null for cases in which trap attempts to remove a nonexistent temporary file. In those cases rm sends an error message to standard error; because standard error is redirected, the user does not see this message.

See page 543 for another example that uses trap.

kill: Aborts a Process

The kill builtin sends a signal to a process or job. The kill command has the following syntax:

kill [signal] PID

where signal is the signal name or number (for example, INT or 2) and PID is the process identification number of the process that is to receive the signal. You can specify a job number (page 131) as %n in place of PID. If you omit signal, kill sends a TERM (software termination, number 15) signal. For more information on signal names and numbers see Table 13-5 on page 577.

The following command sends the TERM signal to job number 1:

$ kill -TERM %1

Because TERM is the default signal for kill, you can also give this command as kill %1. Give the command kill l (lowercase "l") to display a list of signal names.

A program that is interrupted often leaves matters in an unpredictable state: Temporary files may be left behind (when they are normally removed), and permissions may be changed. A well-written application traps, or detects, signals and cleans up before exiting. Most carefully written applications trap the INT, QUIT, and TERM signals.

To terminate a program, first try INT (press CONTROL-C, if the job is in the foreground). Because an application can be written to ignore these signals, you may need to use the KILL signal, which cannot be trapped or ignored; it is a "sure kill." Refer to page 761 for more information on kill. See also the related utility killall (page 763).

getopts: Parses Options

The getopts builtin (not available in tcsh) parses command line arguments, thereby making it easier to write programs that follow the Mac OS X argument conventions. The syntax for getopts is

getopts optstring varname [arg...]

where optstring is a list of the valid option letters, varname is the variable that receives the options one at a time, and arg is the optional list of parameters to be processed. If arg is not present, getopts processes the command line arguments. If optstring starts with a colon (:), the script takes care of generating error messages; otherwise, getopts generates error messages.

The getopts builtin uses the OPTIND (option index) and OPTARG (option argument) variables to store option-related values. When a shell script starts, the value of OPTIND is 1. Each time getopts locates an argument, it increments OPTIND to the index of the next option to be processed. If the option takes an argument, bash assigns the value of the argument to OPTARG.

To indicate that an option takes an argument, follow the corresponding letter in optstring with a colon (:). The option string dxo:lt:r indicates that getopts should search for d, x, o, l, t, and r options and that the o and t options take arguments.

Using getopts as the test-command in a while control structure allows you to loop over the options one at a time. The getopts builtin checks the option list for options that are in optstring. Each time through the loop, getopts stores the option letter it finds in varname.

Suppose that you want to write a program that can take three options:

  1. A b option indicates that the program should ignore whitespace at the start of input lines.

  2. A t option followed by the name of a directory indicates that the program should use that directory for temporary files. Otherwise, it should use /tmp.

  3. A u option indicates that the program should translate all its output to uppercase.

In addition, the program should ignore all other options and end option processing when it encounters two hyphens (--).

The problem is to write the portion of the program that determines which options the user has supplied. The following solution does not use getopts.

SKIPBLANKS=
TMPDIR=/tmp
CASE=lower
while [[ "$1" = -* ]] # [[ = ]] does pattern match
do
    case $1 in
        -b)    SKIPBLANKS=TRUE ;;
        -t)    if [ -d "$2" ]
                   then
                   TMPDIR=$2
                   shift
               else
                   echo "$0: -t takes a directory argument." >&2
                   exit 1
               fi ;;
        -u)    CASE=upper ;;
        --)    break ;; # Stop processing options
        *)     echo "$0: Invalid option $1 ignored." >&2 ;;
        esac
    shift
done

This program fragment uses a loop to check and shift arguments while the argument is not . As long as the argument is not two hyphens, the program continues to loop through a case statement that checks for possible options. The case label breaks out of the while loop. The * case label recognizes any option; it appears as the last case label to catch any unknown options, displays an error message, and allows processing to continue. On each pass through the loop, the program does a shift to get to the next argument. If an option takes an argument, the program does an extra shift to get past that argument.

The following program fragment processes the same options, but uses getopts:

SKIPBLANKS=
TMPDIR=/tmp
CASE=lower

while getopts :bt:u arg
do
    case $arg in
        b)     SKIPBLANKS=TRUE ;;
        t)     if [ -d "$OPTARG" ]
                   then
                   TMPDIR=$OPTARG
               else
                   echo "$0: $OPTARG is not a directory." >&2
                   exit 1
               fi ;;
        u)     CASE=upper ;;
        :)     echo "$0: Must supply an argument to -$OPTARG." >&2
               exit 1 ;;
        \?)    echo "Invalid option -$OPTARG ignored." >&2 ;;
        esac
done

In this version of the code, the while structure evaluates the getopts builtin each time it comes to the top of the loop. The getopts builtin uses the OPTIND variable to keep track of the index of the argument it is to process the next time it is called. There is no need to call shift in this example.

In the getopts version of the script the case patterns do not start with a hyphen because the value of arg is just the option letter (getopts strips off the hyphen). Also, getopts recognizes as the end of the options, so you do not have to specify it explicitly as in the case statement in the first example.

Because you tell getopts which options are valid and which require arguments, it can detect errors in the command line and handle them in two ways. This example uses a leading colon in optstring to specify that you check for and handle errors in your code; when getopts finds an invalid option, it sets varname to ? and OPTARG to the option letter. When it finds an option that is missing an argument, getopts sets varname to : and OPTARG to the option lacking an argument.

The \? case pattern specifies the action to take when getopts detects an invalid option. The : case pattern specifies the action to take when getopts detects a missing option argument. In both cases getopts does not write any error message; it leaves that task to you.

If you omit the leading colon from optstring, both an invalid option and a missing option argument cause varname to be assigned the string ?. OPTARG is not set and getopts writes its own diagnostic message to standard error. Generally this method is less desirable because you have less control over what the user sees when an error is made.

Using getopts will not necessarily make your programs shorter. Its principal advantages are that it provides a uniform programming interface and it enforces standard option handling.

A Partial List of Builtins

Table 13-6 lists some of the bash builtins. See "Listing bash builtins" on page 138 for instructions on how to display complete lists of builtins.

Table 13-6. bash builtins

Builtin

Function

:

Returns 0 or true (the null builtin; page 579)

.(dot)

Executes a shell script as part of the current process (page 261)

bg

Puts a suspended job in the background (page 274)

break

Exits from a looping control structure (page 543)

cd

Changes to another working directory (page 82)

continue

Starts with the next iteration of a looping control structure (page 543)

echo

Displays its arguments (page 52)

eval

Scans and evaluates the command line (page 374)

exec

Executes a shell script or program in place of the current process (page 574)

exit

Exits from the current shell (usually the same as CONTROL-D from an interactive shell; page 564)

export

Places the value of a variable in the calling environment (makes it global; page 560)

fg

Brings a job from the background into the foreground (page 273)

getopts

Parses arguments to a shell script (page 581)

jobs

Displays list of background jobs (page 273)

kill

Sends a signal to a process or job (page 761)

pwd

Displays the name of the working directory (page 77)

read

Reads a line from standard input (page 571)

readonly

Declares a variable to be readonly (page 282)

set

Sets shell flags or command line argument variables; with no argument, lists all variables (pages 318, 353, and 568)

shift

Promotes each command line argument (page 567)

test

Compares arguments (pages 525 and 871)

times

Displays total times for the current shell and its children

trap

Traps a signal (page 577)

type

Displays how each argument would be interpreted as a command (page 570)

umask

Returns the value of the file-creation mask (page 883)

unset

Removes a variable or function (page 282)

wait

Waits for a background process to terminate (page 377)

Expressions

An expression is composed of constants, variables, and operators that can be processed to return a value. This section covers arithmetic, logical, and conditional expressions as well as operators. Table 13-8 on page 588 lists the bash operators.

Arithmetic Evaluation

The Bourne Again Shell can perform arithmetic assignments and evaluate many different types of arithmetic expressions, all using integers. The shell performs arithmetic assignments in a number of ways. One is with arguments to the let builtin:

$ let "VALUE=VALUE * 10 + NEW"

In the preceding example, the variables VALUE and NEW contain integer values. Within a let statement you do not need to use dollar signs ($) in front of variable names. Double quotation marks must enclose a single argument, or expression, that contains SPACEs. Because most expressions contain SPACEs and need to be quoted, bash accepts ((expression)) as a synonym for let "expression", obviating the need for both quotation marks and dollar signs:

$ ((VALUE=VALUE * 10 + NEW))

You can use either form wherever a command is allowed and can remove the SPACEs if you like. In the following example, the asterisk (*) does not need to be quoted because the shell does not perform pathname expansion on the right side of an assignment (page 280):

$ let VALUE=VALUE*10+NEW

Because each argument to let is evaluated as a separate expression, you can assign values to more than one variable on a single line:

$ let "COUNT = COUNT + 1" VALUE=VALUE*10+NEW

You need to use commas to separate multiple assignments within a set of double parentheses:

$ ((COUNT = COUNT + 1, VALUE=VALUE*10+NEW))

Tip: Arithmetic evaluation versus arithmetic expansion

Arithmetic evaluation differs from arithmetic expansion. As explained on page 325, arithmetic expansion uses the syntax $((expression)), evaluates expression, and replaces $((expression)) with the result. You can use arithmetic expansion to display the value of an expression or to assign that value to a variable.

Arithmetic evaluation uses the let expression or ((expression)) syntax, evaluates expression, and returns a status code. You can use arithmetic evaluation to perform a logical comparison or an assignment.

Logical expressions

You can use the ((expression)) syntax for logical expressions, although that task is frequently left to [[expression ]]. The next example expands the age_check script (page 326) to include logical arithmetic evaluation in addition to arithmetic expansion.

$ cat age2
#!/bin/bash
echo -n "How old are you? "
read age
if ((30 < age && age < 60)); then
        echo "Wow, in $((60-age)) years, you'll be 60!"
    else
        echo "You are too young or too old to play."
fi

$ age2
How old are you? 25
You are too young or too old to play.

The test-statement for the if structure evaluates two logical comparisons joined by a Boolean AND and returns 0 (true) if they are both true or 1 (false) otherwise.

Logical Evaluation (Conditional Expressions)

The syntax of a conditional expression is

[[ expression ]]

where expression is a Boolean (logical) expression. You must precede a variable name with a dollar sign ($) within expression. The result of executing this builtin, like the test builtin, is a return status. The conditions allowed within the brackets are almost a superset of those accepted by test (page 871). Where the test builtin uses a as a Boolean AND operator, [[ expression ]] uses &&. Similarly, where test uses o as a Boolean OR operator, [[ expression ]] uses ||.

You can replace the line that tests age in the age2 script (preceding) with the following conditional expression. You must surround the [[ and ]] tokens with whitespace or a command terminator, and place dollar signs before the variables:

if [[ 30 < $age && $age < 60 ]]; then

You can also use test's relational operators gt, ge, lt, le, eq, and ne:

if  [[ 30 -lt $age && $age -lt 60 ]]; then

String comparisons

The test builtin tests whether strings are equal or unequal. The [[ expression ]] syntax adds comparison tests for string operators. The > and < operators compare strings for order (for example, "aa" < "bbb"). The = operator tests for pattern match, not just equality: [[ string = pattern ]] is true if string matches pattern. This operator is not symmetrical; the pattern must appear on the right side of the equal sign. For example, [[ artist = a* ]] is true (= 0), whereas [[ a* = artist ]] is false (= 1):

$ [[ artist = a* ]]
$ echo $?
0
$ [[ a* = artist ]]
$ echo $?
1

The next example uses a command list that starts with a compound condition. The condition tests that the directory bin and the file src/myscript.bash exist. If this is true, cp copies src/myscript.bash to bin/myscript. If the copy succeeds, chmod makes myscript executable. If any of these steps fails, echo displays a message.

$ [[ -d bin && -f src/myscript.bash ]] && cp src/myscript.bash \
bin/myscript && chmod +x bin/myscript || echo "Cannot make \
executable version of myscript"

String Pattern Matching

The Bourne Again Shell provides string pattern-matching operators that can manipulate pathnames and other strings. These operators can delete from strings prefixes or suffixes that match patterns. The four operators are listed in Table 13-7.

Table 13-7. String operators

Operator

Function

#

Removes minimal matching prefixes

##

Removes maximal matching prefixes

%

Removes minimal matching suffixes

%%

Removes maximal matching suffixes

The syntax for these operators is

${varname op pattern}

where op is one of the operators listed in Table 13-7 and pattern is a match pattern similar to that used for filename generation. These operators are commonly used to manipulate pathnames so as to extract or remove components or to change suffixes:

$ SOURCEFILE=/usr/local/src/prog.c
$ echo ${SOURCEFILE#/*/}
local/src/prog.c
$ echo ${SOURCEFILE##/*/}
prog.c
$ echo ${SOURCEFILE%/*}
/usr/local/src
$ echo ${SOURCEFILE%%/*}

$ echo ${SOURCEFILE%.c}
/usr/local/src/prog
$ CHOPFIRST=${SOURCEFILE#/*/}
$ echo $CHOPFIRST
local/src/prog.c
$ NEXT=${CHOPFIRST%%/*}
$ echo $NEXT
local

Here the string-length operator, ${#name}, is replaced by the number of characters in the value of name:

$ echo $SOURCEFILE
/usr/local/src/prog.c
$ echo ${#SOURCEFILE}
21

Operators

Arithmetic expansion and arithmetic evaluation use the same syntax, precedence, and associativity of expressions as the C language. Table 13-8 lists operators in order of decreasing precedence (priority of evaluation); each group of operators has equal precedence. Within an expression you can use parentheses to change the order of evaluation.

Table 13-8. Operators

Type of operator/operator

Function

Post

var++

Postincrement

var

Postdecrement

Pre

++var

Preincrement

var

Predecrement

Unary

Unary minus

+

Unary plus

Negation

!

Boolean NOT (logical negation)

~

Complement (bitwise negation)

Exponentiation

**

Exponent

Multiplication, division, remainder

*

Multiplication

/

Division

%

Remainder

Addition, subtraction

Subtraction

+

Addition

Bitwise shifts

<<

Left bitwise shift

>>

Right bitwise shift

Comparison

<=

Less than or equal

>=

Greater than or equal

<

Less than

>

Greater than

Equality, inequality

==

Equality

!=

Inequality

Bitwise

&

Bitwise AND

^

Bitwise XOR (exclusive OR)

|

Bitwise OR

Boolean (logical)

&&

Boolean AND

||

Boolean OR

Conditional evaluation

?:

Ternary operator

Assignment

=, *=, /=, %=, +=, =, <<=, >>=, &=, ^=, |=

Assignment

Comma

,

Comma

Pipe

The pipe token has higher precedence than operators. You can use pipes anywhere in a command that you can use simple commands. For example, the command line

$ cmd1 | cmd2 || cmd3 | cmd4 && cmd5 | cmd6

is interpreted as if you had typed

$ ((cmd1 | cmd2) || (cmd3 | cmd4)) && (cmd5 | cmd6)

Tip: Do not rely on rules of precedence: use parentheses

Do not rely on the precedence rules when you use compound commands. Instead, use parentheses to explicitly state the order in which you want the shell to interpret the commands.

Increment and decrement operators

The postincrement, postdecrement, preincrement, and predecrement operators work with variables. The pre- operators, which appear in front of the variable name as in ++COUNT and VALUE, first change the value of the variable (++ adds 1; subtracts 1) and then provide the result for use in the expression. The post- operators appear after the variable name as in COUNT++ and VALUE; they first provide the unchanged value of the variable for use in the expression and then change the value of the variable.

$ N=10
$ echo $N
10
$ echo $((--N+3))
12
$ echo $N
9
$ echo $((N++ - 3))
6
$ echo $N
10

Remainder

The remainder operator (%) gives the remainder when its first operand is divided by its second. For example, the expression $((15%7)) has the value 1.

Boolean

The result of a Boolean operation is either 0 (false) or 1 (true).

The && (AND) and || (OR) Boolean operators are called short-circuiting operators. If the result of using one of these operators can be decided by looking only at the left operand, the right operand is not evaluated. The && operator causes the shell to test the exit status of the command preceding it. If the command succeeded, bash executes the next command; otherwise, it skips the remaining commands on the command line. You can use this construct to execute commands conditionally:

$ mkdir bkup && cp -r src bkup

This compound command creates the directory bkup. If mkdir succeeds, the contents of directory src is copied recursively to bkup.

The || separator also causes bash to test the exit status of the first command but has the opposite effect: The remaining command(s) are executed only if the first one failed (that is, exited with nonzero status):

$ mkdir bkup || echo "mkdir of bkup failed" >> /tmp/log

The exit status of a command list is the exit status of the last command in the list. You can group lists with parentheses. For example, you could combine the previous two examples as

$ (mkdir bkup && cp -r src bkup) || echo "mkdir failed" >> /tmp/log

In the absence of parentheses, && and || have equal precedence and are grouped from left to right. The following examples use the true and false utilities. These utilities do nothing and return true (0) and false (1) exit statuses, respectively:

$ false; echo $?
1

The $? variable holds the exit status of the preceding command (page 564). The next two commands yield an exit status of 1 (false):

$ true || false && false
$ echo $?
1
$ (true || false) && false
$ echo $?
1

Similarly the next two commands yield an exit status of 0 (true):

$ false && false || true
$ echo $?
0
$ (false && false) || true
$ echo $?
0

Because || and && have equal precedence, the parentheses in the two preceding pairs of examples do nothing to change the order of operations.

Because the expression on the right side of a short-circuiting operator may never get executed, you must be careful with assignment statements in that location. The following example demonstrates what can happen:

$ ((N=10,Z=0))
$ echo $((N || ((Z+=1)) ))
1
$ echo $Z
0

Because the value of N is nonzero, the result of the || (OR) operation is 1 (true), no matter what the value of the right side is. As a consequence ((Z+=1)) is never evaluated and Z is not incremented.

Ternary

The ternary operator, ? :, decides which of two expressions should be evaluated, based on the value returned from a third expression:

expression1 ? expression2 : expression3

If expression1 produces a false (0) value, expression3 is evaluated; otherwise, expression2 is evaluated. The value of the entire expression is the value of expression2 or expression3, depending on which one is evaluated. If expression1 is true, expression3 is not evaluated. If expression1 is false expression2 is not evaluated:

$ ((N=10,Z=0,COUNT=1))
$ ((T=N>COUNT?++Z:--Z))
$ echo $T
1
$ echo $Z
1

Assignment

The assignment operators, such as +=, are shorthand notations. For example, N+=3 is the same as ((N=N+3)).

Other bases

The following commands use the syntax base#n to assign base 2 (binary) values. First v1 is assigned a value of 0101 (5 decimal) and v2 is assigned a value of 0110 (6 decimal). The echo utility verifies the decimal values.

$ ((v1=2#0101))
$ ((v2=2#0110))
$ echo "$v1 and $v2"
5 and 6

Next the bitwise AND operator (&) selects the bits that are on in both 5 (0101 binary) and 6 (0110 binary). The result is binary 0100, which is 4 decimal.

$ echo $(( v1 & v2 ))
4

The Boolean AND operator (&&) produces a result of 1 if both of its operands are nonzero and a result of 0 otherwise. The bitwise inclusive OR operator (|) selects the bits that are on in either 0101 or 0110, resulting in 0111, which is 7 decimal. The Boolean OR operator (||) produces a result of 1 if either of its operands is nonzero and a result of 0 otherwise.

$ echo $(( v1 && v2 ))
1
$ echo $(( v1 | v2 ))
7
$ echo $(( v1 || v2 ))
1

Next the bitwise exclusive OR operator (^) selects the bits that are on in either, but not both, of the operands 0101 and 0110, yielding 0011, which is 3 decimal. The Boolean NOT operator (!) produces a result of 1 if its operand is 0 and a result of 0 otherwise. Because the exclamation point in $(( ! v1 )) is enclosed within double parentheses, it does not need to be escaped to prevent the shell from interpreting the exclamation point as a history event. The comparison operators produce a result of 1 if the comparison is true and a result of 0 otherwise.

$ echo $(( v1 ^ v2 ))
3
$ echo $(( ! v1 ))
0
$ echo $(( v1 < v2 ))
1
$ echo $(( v1 > v2 ))
0

Shell Programs

The Bourne Again Shell has many features that make it a good programming language. The structures that bash provides are not a random assortment. Rather, they have been chosen to provide most of the structural features that are in other procedural languages, such as C or Pascal. A procedural language provides the ability to

  • Declare, assign, and manipulate variables and constant data. The Bourne Again Shell provides string variables, together with powerful string operators, and integer variables, along with a complete set of arithmetic operators.

  • Break large problems into small ones by creating subprograms. The Bourne Again Shell allows you to create functions and call scripts from other scripts. Shell functions can be called recursively; that is, a Bourne Again Shell function can call itself. You may not need to use recursion often, but it may allow you to solve some apparently difficult problems with ease.

  • Execute statements conditionally, using statements such as if.

  • Execute statements iteratively, using statements such as while and for.

  • Transfer data to and from the program, communicating with both data files and users.

Programming languages implement these capabilities in different ways but with the same ideas in mind. When you want to solve a problem by writing a program, you must first figure out a procedure that leads you to a solutionthat is, an algorithm. Typically you can implement the same algorithm in roughly the same way in different programming languages, using the same kinds of constructs in each language.

Chapter 8 and this chapter have introduced numerous bash features, many of which are useful for interactive use as well as for shell programming. This section develops two complete shell programs, demonstrating how to combine some of these features effectively. The programs are presented as problems for you to solve along with sample solutions.

A Recursive Shell Script

A recursive construct is one that is defined in terms of itself. Alternatively, you might say that a recursive program is one that can call itself. This may seem circular, but it need not be. To avoid circularity a recursive definition must have a special case that is not self-referential. Recursive ideas occur in everyday life. For example, you can define an ancestor as your mother, your father, or one of their ancestors. This definition is not circular; it specifies unambiguously who your ancestors are: your mother or your father, or your mother's mother or father or your father's mother or father, and so on.

A number of Mac OS X system utilities can operate recursively. See the R option to the chmod (page 676), chown (page 682), and cp (page 690) utilities for examples.

Solve the following problem by using a recursive shell function:

Write a shell function named makepath that, given a pathname, creates all components in that pathname as directories. For example, the command makepath a/b/c/d should create directories a, a/b, a/b/c, and a/b/c/d. (The mkdir utility supports a p option that does exactly this. Solve the problem without using mkdir p.)

One algorithm for a recursive solution follows:

  1. Examine the path argument. If it is a null string or if it names an existing directory, do nothing and return.

  2. If it is a simple path component, create it (using mkdir) and return.

  3. Otherwise, call makepath using the path prefix of the original argument. This step eventually creates all the directories up to the last component, which you can then create with mkdir.

In general, a recursive function must invoke itself with a simpler version of the problem than it was given until it is finally called with a simple case that does not need to call itself. Following is one possible solution based on this algorithm:

makepath

# this is a function
# enter it at the keyboard, do not run it as a shell script
#
function makepath()
{
    if [[ ${#1} -eq 0 || -d "$1" ]]
        then
            return 0       # Do nothing
    fi
    if [[ "${1%/*}" = "$1" ]]
        then
            mkdir $1
            return $?
    fi
    makepath ${1%/*} || return 1
    mkdir $1
    return $?
}

In the test for a simple component (the if statement in the middle of the function), the left expression is the argument after the shortest suffix that starts with a / character has been stripped away (page 587). If there is no such character (for example, if $1 is alex), nothing is stripped off and the two sides are equal. If the argument is a simple filename preceded by a slash, such as /usr, the expression ${1%/*} evaluates to a null string. To make the function work in this case, you must take two precautions: Put the left expression within quotation marks and ensure that the recursive function behaves sensibly when it is passed a null string as an argument. In general, good programs are robust: They should be prepared for borderline, invalid, or meaningless input and behave appropriately in such cases.

By giving the following command from the shell you are working in, you turn on debugging tracing so that you can watch the recursion work:

$ set -o xtrace

(Give the same command, but replace the hyphen with a plus sign (+) to turn debugging off.) With debugging turned on, the shell displays each line in its expanded form as it executes the line. A + precedes each line of debugging output. In the following example, the first line that starts with + shows the shell calling makepath. The makepath function is called from the command line with arguments of a/b/c. Subsequently it calls itself with arguments of a/b and finally a. All the work is done (using mkdir) as each call to makepath returns.

$ makepath a/b/c
+ makepath a/b/c
+ [[ 5 -eq 0 ]]
+ [[ -d a/b/c ]]
+ [[ a/b = \a\/\b\/\c ]]
+ makepath a/b
+ [[ 3 -eq 0 ]]
+ [[ -d a/b ]]
+ [[ a = \a\/\b ]]
+ makepath a
+ [[ 1 -eq 0 ]]
+ [[ -d a ]]
+ [[ a = \a ]]
+ mkdir a
+ return 0
+ mkdir a/b
+ return 0
+ mkdir a/b/c
+ return 0

The function works its way down the recursive path and back up again.

It is instructive to invoke makepath with an invalid path and see what happens. The following example, run with debugging turned on, tries to create the path /a/b, which requires that you create directory a in the root directory. Unless you have permission to write to the root directory, you are not permitted to create this directory.

$ makepath /a/b
+ makepath /a/b
+ [[ 4 -eq 0 ]]
+ [[ -d /a/b ]]
+ [[ /a = \/\a\/\b ]]
+ makepath /a
+ [[ 2 -eq 0 ]]
+ [[ -d /a ]]
+ [[ '' = \/\a ]]
+ makepath
+ [[ 0 -eq 0 ]]
+ return 0
+ mkdir /a
mkdir: cannot create directory '/a': Permission denied
+ return 1
+ return 1

The recursion stops when makepath is denied permission to create the /a directory. The error return is passed all the way back, so the original makepath exits with nonzero status.

Tip: Use local variables with recursive functions

The preceding example glossed over a potential problem that you may encounter when you use a recursive function. During the execution of a recursive function, many separate instances of that function may be active simultaneously. All but one of them are waiting for their child invocation to complete.

Because functions run in the same environment as the shell that calls them, variables are implicitly shared by a shell and a function it calls so that all instances of the function share a single copy of each variable. Sharing variables can give rise to side effects that are rarely what you want. As a rule, you should use typeset to make all variables of a recursive function be local variables. See page 561 for more information.

The quiz Shell Script

Solve the following problem using a bash script:

Write a generic multiple-choice quiz program. The program should get its questions from data files, present them to the user, and keep track of the number of correct and incorrect answers. The user must be able to exit from the program at any time with a summary of results to that point.

The detailed design of this program and even the detailed description of the problem depend on a number of choices: How will the program know which subjects are available for quizzes? How will the user choose a subject? How will the program know when the quiz is over? Should the program present the same questions (for a given subject) in the same order each time, or should it scramble them?

Of course, you can make many perfectly good choices that implement the specification of the problem. The following details narrow the problem specification:

  • Each subject will correspond to a subdirectory of a master quiz directory. This directory will be named in the environment variable QUIZDIR, whose default will be ~/quiz. For example, you could have the following directories correspond to the subjects engineering, art, and politics: ~/quiz/engineering, ~/quiz/art, and ~/quiz/politics.

  • Each subject can have several questions. Each question is represented by a file in its subject's directory.

  • The first line of each file that represents a question is the text of the question. If it takes more than one line, you must escape the NEWLINE with a backslash. (This setup makes it easy to read a single question with the read builtin.) The second line of the file is an integer that specifies the number of choices. The next lines are the choices themselves. The last line is the correct answer. Following is a sample question file:

    Who discovered the principle of the lever?
4
Euclid
Archimedes
Thomas Edison
The Lever Brothers
Archimedes

  • The program presents all the questions in a subject directory. At any point the user can interrupt the quiz with CONTROL-C, whereupon the program will summarize the results so far and exit. If the user does not interrupt, the program summarizes the results and exits when it has asked all questions for the chosen subject.

  • The program scrambles the questions in a subject before presenting them.

Following is a top-level design for this program:

  1. Initialize. This involves a number of steps, such as setting the counts of the number of questions asked so far and the number of correct and wrong answers to zero. Sets up to trap CONTROL-C.

  2. Present the user with a choice of subjects and get the user's response.

  3. Change to the corresponding subject directory.

  4. Determine the questions to be asked (that is, the filenames in that directory). Arrange them in random order.

  5. Repeatedly present questions and ask for answers until the quiz is over or is interrupted by the user.

  6. Present the results and exit.

Clearly some of these steps (such as step 3) are simple, whereas others (such as step 4) are complex and worthy of analysis on their own. Use shell functions for any complex step, and use the trap builtin to handle a user interrupt.

Here is a skeleton version of the program with empty shell functions:

function initialize
{
# Initializes variables.
}
function choose_subj
{
# Writes choice to standard output.
}

function scramble
{
# Stores names of question files, scrambled,
# in an array variable named questions.
}

function ask
{
# Reads a question file, asks the question, and checks the
# answer. Returns 1 if the answer was correct, 0 otherwise. If it
# encounters an invalid question file, exit with status 2.
}

function summarize
{
# Presents the user's score.
}

# Main program
initialize                       # Step 1 in top-level design

subject=$(choose_subj)           # Step 2
[[ $? -eq 0 ]] || exit 2         # If no valid choice, exit

cd $subject || exit 2            # Step 3
echo                             # Skip a line
scramble                         # Step 4

for ques in ${questions[*]}; do  # Step 5
    ask $ques
    result=$?
    (( num_ques=num_ques+1 ))
    if [[ $result == 1 ]]; then
        (( num_correct += 1 ))
    fi
    echo                         # Skip a line between questions
    sleep ${QUIZDELAY:=1}
done

summarize                        # Step 6
exit 0

To make reading the results a bit easier for the user, a sleep call appears inside the question loop. It delays $QUIZDELAY seconds (default = 1) between questions.

Now the task is to fill in the missing pieces of the program. In a sense this program is being written backward. The details (the shell functions) come first in the file but come last in the development process. This common programming practice is called top-down design. In top-down design you fill in the broad outline of the program first and supply the details later. In this way you break the problem up into smaller problems, each of which you can work on independently. Shell functions are a great help in using the top-down approach.

One way to write the initialize function follows. The cd command causes QUIZDIR to be the working directory for the rest of the script and defaults to ~/quiz if QUIZDIR is not set.

function initialize ()
{
trap 'summarize ; exit 0' INT    # Handle user interrupts
num_ques=0                       # Number of questions asked so far
num_correct=0                    # Number answered correctly so far
first_time=true                  # true until first question is asked
cd ${QUIZDIR:=~/quiz} || exit 2
}

Be prepared for the cd command to fail. The directory may be unsearchable or conceivably another user may have removed it. The preceding function exits with a status code of 2 if cd fails.

The next function, choose_subj, is a bit more complicated. It displays a menu using a select statement:

function choose_subj ()
{
subjects=($(ls))
PS3="Choose a subject for the quiz from the preceding list: "
select Subject in ${subjects[*]}; do
    if [[ -z "$Subject" ]]; then
        echo "No subject chosen. Bye." >&2
        exit 1
    fi
    echo $Subject
    return 0
done
}

The function first uses an ls command and command substitution to put a list of subject directories in the subjects array. Next the select structure (page 551) presents the user with a list of subjects (the directories found by ls) and assigns the chosen directory name to the Subject variable. Finally the function writes the name of the subject directory to standard output. The main program uses command substitution to assign this value to the subject variable [subject=$(choose_subj)].

The scramble function presents a number of difficulties. In this solution it uses an array variable (questions) to hold the names of the questions. It scrambles the entries in an array using the RANDOM variable (each time you reference RANDOM it has the value of a [random] integer between 0 and 32767):

function scramble ()
{
typeset -i index quescount
questions=($(ls))
quescount=${#questions[*]}         # Number of elements
((index=quescount-1))
while [[ $index > 0 ]]; do
    ((target=RANDOM % index))
    exchange $target $index
    ((index -= 1))
done
}

This function initializes the array variable questions to the list of filenames (questions) in the working directory. The variable quescount is set to the number of such files. Then the following algorithm is used: Let the variable index count down from quescount 1 (the index of the last entry in the array variable). For each value of index, the function chooses a random value target between 0 and index, inclusive. The command

((target=RANDOM % index))

produces a random value between 0 and index 1 by taking the remainder (the % operator) when $RANDOM is divided by index. The function then exchanges the elements of questions at positions target and index. It is convenient to do this in another function named exchange:

function exchange ()
{
temp_value=${questions[$1]}
questions[$1]=${questions[$2]}
questions[$2]=$temp_value
}

The ask function also uses the select structure. It reads the question file named in its argument and uses the contents of that file to present the question, accept the answer, and determine whether the answer is correct. (See the code that follows.)

The ask function uses file descriptor 3 to read successive lines from the question file, whose name was passed as an argument and is represented by $1 in the function. It reads the question into the ques variable and the number of questions into num_opts. The function constructs the variable choices by initializing it to a null string and successively appending the next choice. Then it sets PS3 to the value of ques and uses a select structure to prompt the user with ques. The select structure places the user's answer in answer, and the function then checks it against the correct answer from the file.

The construction of the choices variable is done with an eye toward avoiding a potential problem. Suppose that one answer has some whitespace in it. Then it might appear as two or more arguments in choices. To avoid this problem, make sure that choices is an array variable. The select statement does the rest of the work:

quiz

$ cat quiz
#!/bin/bash

# remove the # on the following line to turn on debugging
# set -o xtrace

#==================
function initialize ()
{
trap 'summarize ; exit 0' INT     # Handle user interrupts
num_ques=0                        # Number of questions asked so far
num_correct=0                     # Number answered correctly so far
first_time=true                   # true until first question is asked
cd ${QUIZDIR:=~/quiz} || exit 2
}

#==================
function choose_subj ()
{
subjects=($(ls))
PS3="Choose a subject for the quiz from the preceding list: "
select Subject in ${subjects[*]}; do
    if [[ -z "$Subject" ]]; then
        echo "No subject chosen. Bye." >&2
        exit 1
    fi
    echo $Subject
    return 0
done
}

#==================
function exchange ()
{
temp_value=${questions[$1]}
questions[$1]=${questions[$2]}
questions[$2]=$temp_value
}

#==================
function scramble ()
{
typeset -i index quescount
questions=($(ls))
quescount=${#questions[*]}        # Number of elements
((index=quescount-1))
while [[ $index > 0 ]]; do
    ((target=RANDOM % index))
    exchange $target $index
    ((index -= 1))
done
}
#==================
function ask ()
{
exec 3<$1
read -u3 ques || exit 2
read -u3 num_opts || exit 2

index=0
choices=()
while (( index < num_opts )) ; do
    read -u3 next_choice || exit 2
    choices=("${choices[@]}" "$next_choice")
    ((index += 1))
done
read -u3 correct_answer || exit 2
exec 3<&-

if [[ $first_time = true ]]; then
    first_time=false
    echo -e "You may press the interrupt key at any time to quit.\n"
fi

PS3=$ques"  "                     # Make $ques the prompt for select
                                  # and add some spaces for legibility.
select answer in "${choices[@]}"; do
    if [[ -z "$answer" ]]; then
            echo  Not a valid choice. Please choose again.
        elif [[ "$answer" = "$correct_answer" ]]; then
            echo "Correct!"
            return 1
        else
            echo "No, the answer is $correct_answer."
            return 0
    fi
done
}

#==================
function summarize ()
{
echo                              # Skip a line
if (( num_ques == 0 )); then
    echo "You did not answer any questions"
    exit 0
fi

(( percent=num_correct*100/num_ques ))
echo "You answered $num_correct questions correctly, out of \
$num_ques total questions."
echo "Your score is $percent percent."
}
#==================
# Main program
initialize                        # Step 1 in top-level design

subject=$(choose_subj)            # Step 2
[[ $? -eq 0 ]] || exit 2          # If no valid choice, exit

cd $subject || exit 2             # Step 3
echo                              # Skip a line
scramble                          # Step 4

for ques in ${questions[*]}; do   # Step 5
    ask $ques
    result=$?
    (( num_ques=num_ques+1 ))
    if [[ $result == 1 ]]; then
        (( num_correct += 1 ))
    fi
    echo                          # Skip a line between questions
    sleep ${QUIZDELAY:=1}
done

summarize                         # Step 6
exit 0

Chapter Summary

The shell is a programming language. Programs written in this language are called shell scripts, or simply scripts. Shell scripts provide the decision and looping control structures present in high-level programming languages while allowing easy access to system utilities and user programs. Shell scripts can use functions to modularize and simplify complex tasks.

Control structures

The control structures that use decisions to select alternatives are if...then, if...then...else, and if...then...elif. The case control structure provides a multiway branch and can be used when you want to express alternatives using a simple pattern-matching syntax.

The looping control structures are for...in, for, until, and while. These structures perform one or more tasks repetitively.

The break and continue control structures alter control within loops: break transfers control out of a loop, and continue transfers control immediately to the top of a loop.

The Here document allows input to a command in a shell script to come from within the script itself.

File descriptors

The Bourne Again Shell provides the ability to manipulate file descriptors. Coupled with the read and echo builtins, file descriptors allow shell scripts to have as much control over input and output as programs written in lower-level languages.

Variables

You assign attributes, such as readonly, to bash variables using the typeset builtin. The Bourne Again Shell provides operators to perform pattern matching on variables, provide default values for variables, and evaluate the length of variables. This shell also supports array variables and local variables for functions and provides built-in integer arithmetic capability, using the let builtin and an expression syntax similar to the C programming language.

Builtins

Bourne Again Shell builtins include type, read, exec, trap, kill, and getopts. The type builtin displays information about a command, including its location; read allows a script to accept user input.

The exec builtin executes a command without creating a new process. The new command overlays the current process, assuming the same environment and PID number of that process. This builtin executes user programs and other Mac OS X commands when it is not necessary to return control to the calling process.

The trap builtin catches a signal sent by Mac OS X to the process running the script and allows you to specify actions to be taken upon receipt of one or more signals. You can use this builtin to cause a script to ignore the signal that is sent when the user presses the interrupt key.

The kill builtin allows you to terminate a running program. The getopts builtin parses command line arguments, making it easier to write programs that follow standard conventions for command line arguments and options.

Utilities in scripts

In addition to using control structures, builtins, and functions, shell scripts generally call utilities. The find utility, for instance, is commonplace in shell scripts that search for files in the system hierarchy and can perform a vast range of tasks, from simple to complex.

A well-written shell script adheres to standard programming practices, such as specifying the shell to execute the script on the first line of the script, verifying the number and type of arguments that the script is called with, displaying a standard usage message to report command line errors, and redirecting all informational messages to standard error.

Expressions

There are two basic types of expressions: arithmetic and logical. Arithmetic expressions allow you to do arithmetic on constants and variables, yielding a numeric result. Logical (Boolean) expressions compare expressions or strings, or test conditions to yield a true or false result. As with all decisions within UNIX shell scripts, a true status is represented by the value zero; false, by any nonzero value.

Exercises

1.

Rewrite the journal script of Chapter 8 (question 5, page 334) by adding commands to verify that the user has write permission for a file named journal-file in the user's home directory, if such a file exists. The script should take appropriate actions if journal-file exists and the user does not have write permission to the file. Verify that the modified script works.

2.

The special parameter "$@" is referenced twice in the out script (page 529). Explain what would be different if the parameter "$*" were used in its place.

3.

Write a filter that takes a list of files as input and outputs the basename (page 550) of each file in the list.

4.

Write a function that takes a single filename as an argument and adds execute permission to the file for the user.

  1. When might such a function be useful?

  2. Revise the script so that it takes one or more filenames as arguments and adds execute permission for the user for each file argument.

  3. What can you do to make the function available every time you log in?

  4. Suppose that, in addition to having the function available on subsequent login sessions, you want to make the function available now in your current shell. How would you do so?

5.

When might it be necessary or advisable to write a shell script instead of a shell function? Give as many reasons as you can think of.

6.

Write a shell script that displays the names of all directory files, but no other types of files, in the working directory.

7.

Write a script to display the time every 15 seconds. Read the date man page and display the time, using the %r field descriptor. Clear the window (using the clear command) each time before you display the time.

8.

Enter the following script named savefiles, and give yourself execute permission to the file:

$ cat savefiles
#! /bin/bash
echo "Saving files in current directory in file savethem."
exec > savethem
for i in *
        do
        echo "==================================================="
        echo "File: $i"
        echo "==================================================="
        cat "$i"
        done

  1. What error message do you get when you execute this script? Rewrite the script so that the error does not occur, making sure the output still goes to savethem.

  2. What might be a problem with running this script twice in the same directory? Discuss a solution to this problem.

9.

Read the bash man or info page, try some experiments, and answer the following questions:

  1. How do you export a function?

  2. What does the hash builtin do?

  3. What happens if the argument to exec is not executable?

10.

Using the find utility, perform the following tasks:

  1. List all files in the working directory and all subdirectories that have been modified within the last day.

  2. List all files that you have read access to on the system that are larger than 1 megabyte.

  3. List the inode numbers of all files in the working directory whose file-names end in .c.

  4. List all files that you have read access to on the root filesystem (startup volume) that have been modified in the last 30 days.

11.

Write a short script that tells you whether the permissions for two files, whose names are given as arguments to the script, are identical. If the permissions for the two files are identical, output the common permission field. Otherwise, output each filename followed by its permission field. (Hint: Try using the cut utility.)

12.

Write a script that takes the name of a directory as an argument and searches the file hierarchy rooted at that directory for zero-length files. Write the names of all zero-length files to standard output. If there is no option on the command line, have the script delete the file after displaying its name, asking the user for confirmation, and receiving positive confirmation. A f (force) option on the command line indicates that the script should display the filename but not ask for confirmation before deleting the file.

Advanced Exercises

13.

Write a script that takes a colon-separated list of items and outputs the items, one per line, to standard output (without the colons).

14.

Generalize the script written in exercise 13 so that the character separating the list items is given as an argument to the function. If this argument is absent, the separator should default to a colon.

15.

Write a function named funload that takes as its single argument the name of a file containing other functions. The purpose of funload is to make all functions in the named file available in the current shell; that is, funload loads the functions from the named file. To locate the file, funload searches the colon-separated list of directories given by the environment variable FUNPATH. Assume that the format of FUNPATH is the same as PATH and that searching FUNPATH is similar to the shell's search of the PATH variable.

16.

Rewrite bundle (page 554) so that the script it creates takes an optional list of filenames as arguments. If one or more filenames are given on the command line, only those files should be re-created; otherwise, all files in the shell archive should be re-created. For example, suppose that all files with the filename extension .c are bundled into an archive named srcshell, and you want to unbundle just the files test1.c and test2.c. The following command will unbundle just these two files:

$ bash srcshell test1.c test2.c

17.

What kind of links will the lnks script (page 532) not find? Why?

18.

In principle, recursion is never necessary. It can always be replaced by an iterative construct, such as while or until. Rewrite makepath (page 593) as a nonrecursive function. Which version do you prefer? Why?

19.

Lists are commonly stored in environment variables by putting a colon (:) between each of the list elements. (The value of the PATH variable is a good example.) You can add an element to such a list by catenating the new element to the front of the list, as in

PATH=/opt/bin:$PATH

If the element you add is already in the list, you now have two copies of it in the list. Write a shell function named addenv that takes two arguments: (1) the name of a shell variable and (2) a string to prepend to the list that is the value of the shell variable only if that string is not already an element of the list. For example, the call

addenv PATH /opt/bin

would add /opt/bin to PATH only if that pathname is not already in PATH. Be sure that your solution works even if the shell variable starts out empty. Also make sure that you check the list elements carefully. If /usr/opt/bin is in PATH but /opt/bin is not, the example just given should still add /opt/bin to PATH. (Hint: You may find this exercise easier to complete if you first write a function locate_field that tells you whether a string is an element in the value of a variable.)

20.

Write a function that takes a directory name as an argument and writes to standard output the maximum of the lengths of all filenames in that directory. If the function's argument is not a directory name, write an error message to standard output and exit with nonzero status.

21.

Modify the function you wrote for exercise 20 to descend all subdirectories of the named directory recursively and to find the maximum length of any filename in that hierarchy.

22.

Write a function that lists the number of ordinary files, directories, block special files, character special files, FIFOs, and symbolic links in the working directory. Do this in two different ways:

  1. Use the first letter of the output of ls l to determine a file's type.

  2. Use the file type condition tests of the [[ expression ]] syntax to determine a file's type.

23.

Modify the quiz program (page 596) so that the choices for a question are randomly arranged.


Chapter 14. The awk Pattern Processing Language

IN THIS CHAPTER

Syntax | 610
Arguments | 610
Options | 611
Patterns | 612
Actions | 612
Variables | 613
Functions | 614
Associative Arrays | 615
Control Structures | 616
Examples | 618
getline: Controlling Input | 634
Error Messages | 637

The awk utility is a pattern-scanning and processing language that searches one or more files to see whether they contain records (usually lines) that match specified patterns. It processes lines by performing actions, such as writing the record to standard output or incrementing a counter, each time it finds a match. As opposed to procedural languages, the awk language is data driven: You describe the data you want to work with and tell awk what to do with the data once it finds it.

You can use awk to generate reports or filter text. It works equally well with numbers and text; when you mix the two, awk usually comes up with the right answer. The authors of awk (Alfred V. Aho, Peter J. Weinberger, and Brian W. Kernighan) designed the original utility to be easy to use. To achieve this end they sacrificed execution speed.

The awk utility takes many of its constructs from the C programming language. It includes the following features:

  • Flexible format

  • Conditional execution

  • Looping statements

  • Numeric variables

  • String variables

  • Regular expressions

  • Relational expressions

  • C's printf


Syntax

An awk command line has the following syntax:

awk [options] [program] [file-list]
awk [options] f program-file [file-list]

The awk utility takes its input from files you specify on the command line or from standard input. An advanced command, getline, gives you more choices about where input comes from and how you read it. Unless you redirect output from awk, it goes to standard output.


Arguments

In the preceding syntax, program is an awk program that you include on the command line. The program-file is the name of the file that holds an awk program. Putting the program on the command line allows you to write short awk programs without having to create a separate program-file. To prevent the shell from interpreting the awk commands as shell commands, enclose the program within single quotation marks. Putting a long or complex program in a file can reduce errors and retyping.

The file-list contains pathnames of the ordinary files that awk processes. These files are the input files. When you do not specify a file-list, awk takes input from standard input or as specified by getline (page 634).


Options

F fs

Uses fs as the value of the input field separator (FS variable).

f program-file

Reads the awk program from the file named program-file instead of the command line. You can specify this option more than once on a command line.

v var=value

Assigns value to the variable var. The assignment takes place prior to execution of the awk program and is available within the BEGIN pattern (page 612). You can specify this option more than once on a command line.


Notes

Many UNIX-like systems use gawk (GNU awk). Scripts written for gawk may not run under awk. You can download and build gawk. The gawk home page is www.gnu.org/software/gawk, and contains links for documentation and source code.

See page 637 for examples of awk error messages and how to interpret them.


Language Basics

An awk program (from the command line or from program-file) consists of one or more lines containing a pattern and/or action in the following format:

pattern { action }

The pattern selects lines from the input. The awk utility performs the action on all lines that the pattern selects. The braces surrounding the action enable awk to differentiate it from the pattern. If a program line does not contain a pattern, awk selects all lines in the input. If a program line does not contain an action, awk copies the selected lines to standard output.

To start, awk compares the first line of input (from the file-list or standard input) with each pattern in the program. If a pattern selects the line (if there is a match), awk takes the action associated with the pattern. If the line is not selected, awk takes no action. When awk has completed its comparisons for the first line of input, it repeats the process for the next line of input, continuing this process of comparing subsequent lines of input until it has read all of the input.

If several patterns select the same line, awk takes the actions associated with each of the patterns in the order in which they appear in the program. It is possible for awk to send a single line from the input to standard output more than once.

Patterns

You can use a regular expression (Appendix A), enclosed within slashes, as a pattern. The ~ operator tests whether a field or variable matches a regular expression. The !~ operator tests for no match. You can perform both numeric and string comparisons using the relational operators listed in Table 14-1. You can combine any of the patterns using the Boolean operators || (OR) or && (AND).

Table 14-1. Relational operators

Relop

Meaning

<

Less than

<=

Less than or equal to

= =

Equal to

!=

Not equal to

>=

Greater than or equal to

>

Greater than

BEGIN and END

Two unique patterns, BEGIN and END, execute commands before awk starts its processing and after it finishes. The awk utility executes the actions associated with the BEGIN pattern before, and with the END pattern after, it processes all the input.

, (comma)

The comma is the range operator. If you separate two patterns with a comma on a single awk program line, awk selects a range of lines, beginning with the first line that matches the first pattern. The last line awk selects is the next subsequent line that matches the second pattern. If no line matches the second pattern, awk selects every line through the end of the input. After awk finds the second pattern, it begins the process again by looking for the first pattern again.

Actions

The action portion of an awk command causes awk to take that action when it matches a pattern. When you do not specify an action, awk performs the default action, which is the print command (explicitly represented as {print}). This action copies the record (normally a linesee "Variables" later in this chapter) from the input to standard output.

When you follow a print command with arguments, awk displays only the arguments you specify. These arguments can be variables or string constants. You can send the output from a print command to a file (>), append it to a file (>>), or send it through a pipe to the input of another program (|).

Unless you separate items in a print command with commas, awk catenates them. Commas cause awk to separate the items with the output field separator (OFS, normally a SPACEsee "Variables").

You can include several actions on one line by separating them with semicolons.

Comments

The awk utility disregards anything on a program line following a pound sign (#). You can document an awk program by preceding comments with this symbol.

Variables

Although you do not need to declare awk variables prior to their use, you can optionally assign initial values to them. Unassigned numeric variables are initialized to 0; string variables are initialized to the null string. In addition to user variables, awk maintains program variables. You can use both user and program variables in the pattern and in the action portion of an awk program. Table 14-2 lists a few program variables.

Table 14-2. Variables

Variable

Meaning

$0

The current record (as a single variable)

$1-$n

Fields in the current record

FILENAME

Name of the current input file (null for standard input)

FS

Input field separator (default: SPACE or TAB)

NF

Number of fields in the current record

NR

Record number of the current record

OFS

Output field separator (default: SPACE)

ORS

Output record separator (default: NEWLINE)

RS

Input record separator (default: NEWLINE)

In addition to initializing variables within a program, you can use the v option to initialize variables on the command line. This feature is useful when the value of a variable changes from one run of awk to the next.

By default the input and output record separators are NEWLINE characters. Thus awk takes each line of input to be a separate record and appends a NEWLINE to the end of each output record. By default the input field separators are SPACEs and TABs. The default output field separator is a SPACE. You can change the value of any of the separators at any time by assigning a new value to its associated variable either from within the program or from the command line by using the v option.

Functions

Table 14-3 lists a few of the functions that awk provides for manipulating numbers and strings.

Table 14-3. Functions

Function

Meaning

length(str)

Returns the number of characters in str; without an argument, returns the number of characters in the current record

int(num)

Returns the integer portion of num

index(str1, str2)

Returns the index of str2 in str1 or 0 if str2 is not present

split(str,arr,del)

Places elements of str, delimited by del, in the array arr[1]...arr[n]; returns the number of elements in the array

sprintf(fmt,args)

Formats args according to fmt and returns the formatted string; mimics the C programming language function of the same name

substr(str,pos,len)

Returns the substring of str that begins at pos and is len characters long

tolower(str)

Returns a copy of str in which all uppercase letters are replaced with their lowercase counterparts

toupper(str)

Returns a copy of str in which all lowercase letters are replaced with their uppercase counterparts

Arithmetic Operators

The awk arithmetic operators listed in Table 14-4 are from the C programming language.

Table 14-4. Arithmetic operators

Operator

Meaning

*

Multiplies the expression preceding the operator by the expression following it

/

Divides the expression preceding the operator by the expression following it

%

Takes the remainder after dividing the expression preceding the operator by the expression following it

+

Adds the expression preceding the operator to the expression following it

Subtracts the expression following the operator from the expression preceding it

=

Assigns the value of the expression following the operator to the variable preceding it

++

Increments the variable preceding the operator

Decrements the variable preceding the operator

+=

Adds the expression following the operator to the variable preceding it and assigns the result to the variable preceding the operator

=

Subtracts the expression following the operator from the variable preceding it and assigns the result to the variable preceding the operator

*=

Multiplies the variable preceding the operator by the expression following it and assigns the result to the variable preceding the operator

/=

Divides the variable preceding the operator by the expression following it and assigns the result to the variable preceding the operator

%=

Assigns the remainder, after dividing the variable preceding the operator by the expression following it, to the variable preceding the operator

Associative Arrays

An associative array is one of awk's most powerful features. These arrays use strings as indexes. Using an associative array, you can mimic a traditional array by using numeric strings as indexes.

You assign a value to an element of an associative array just as you would assign a value to any other awk variable. The syntax is

array[string] = value

where array is the name of the array, string is the index of the element of the array you are assigning a value to, and value is the value you are assigning to that element.

You can use a special for structure with an associative array. The syntax is

for (elem in array) action

where elem is a variable that takes on the value of each element of the array as the for structure loops through them, array is the name of the array, and action is the action that awk takes for each element in the array. You can use the elem variable in this action.

The "Examples" section found later in this chapter contains programs that use associative arrays.

printf

You can use the printf command in place of print to control the format of the output that awk generates. The awk version of printf is similar to that found in the C language. A printf command has the following syntax:

printf "control-string", arg1, arg2, ..., argn

The control-string determines how printf formats arg1, arg2, ..., argn. These arguments can be variables or other expressions. Within the control-string you can use \n to indicate a NEWLINE and \t to indicate a TAB. The control-string contains conversion specifications, one for each argument. A conversion specification has the following syntax:

%[][x[.y]]conv

where causes printf to left-justify the argument; x is the minimum field width, and .y is the number of places to the right of a decimal point in a number. The conv indicates the type of numeric conversion and can be selected from the letters in Table 14-5. Refer to "Examples" later in this chapter for examples of how to use printf.

Table 14-5. Numeric conversion

conv

Type of conversion

d

Decimal

e

Exponential notation

f

Floating-point number

g

Use f or e, whichever is shorter

o

Unsigned octal

s

String of characters

x

Unsigned hexadecimal

Control Structures

Control (flow) statements alter the order of execution of commands within an awk program. This section details the if...else, while, and for control structures. In addition, the break and continue statements work in conjunction with the control structures to alter the order of execution of commands. See page 524 for more information on control structures. You do not need to use braces around commands when you specify a single, simple command.

if...else

The if...else control structure tests the status returned by the condition and transfers control based on this status. The syntax of an if...else structure is shown below. The else part is optional.

if (condition)
         {commands}
     [else
         {commands}]

The simple if statement shown here does not use braces:

if ($5 <= 5000) print $0

Next is an awk program that uses a simple if...else structure. Again, there are no braces.

$ cat if1
BEGIN   {
        nam="sam"
        if (nam == "max")
               print "nam is max"
            else
               print "nam is not max, it is", nam
        }
$ awk -f if1
nam is not max, it is sam

while

The while structure loops through and executes the commands as long as the condition is true. The syntax of a while structure is

while (condition)
    {commands}

The next awk program uses a simple while structure to display powers of 2. This example uses braces because the while loop contains more than one statement.

$ cat while1
BEGIN{
    n = 1
    while (n <= 5)
        {
        print n "^2", 2**n
        n++
        }
    }

$ awk -f while1
1^2 2
2^2 4
3^2 8
4^2 16
5^2 32

for

The syntax of a for control structure is

for (init; condition; increment)
    {commands}

A for structure starts by executing the init statement, which usually sets a counter to 0 or 1. It then loops through the commands as long as the condition is true. After each loop it executes the increment statement. The for1 awk program does the same thing as the preceding while1 program except that it uses a for statement, which makes the program simpler:

$ cat for1
BEGIN   {
        for (n=1; n <= 5; n++)
        print n "^2", 2**n
        }

$ awk -f for1
1^2 2
2^2 4
3^2 8
4^2 16
5^2 32

The awk utility supports an alternative for syntax for working with associative arrays:

for (var in array)
     {commands}

This for structure loops through elements of the associative array named array, assigning the value of the index of each element of array to var each time through the loop.

END    {for (name in manuf) print name, manuf[name]}

break

The break statement transfers control out of a for or while loop, terminating execution of the innermost loop it appears in.

continue

The continue statement transfers control to the end of a for or while loop, causing execution of the innermost loop it appears in to continue with the next iteration.


Examples

cars data file

Many of the examples in this section work with the cars data file. From left to right the columns in the file contain each car's make, model, year of manufacture, mileage in thousands of miles, and price. All whitespace in this file is composed of single TABs (the file does not contain any SPACEs).

$ cat cars
plym    fury    1970    73    2500
chevy   malibu  1999    60    3000
ford    mustang 1965    45    10000
volvo   s80     1998    102   9850
ford    thundbd 2003    15    10500
chevy   malibu  2000    50    3500
bmw     325i    1985    115   450
honda   accord  2001    30    6000
ford    taurus  2004    10    17000
toyota  rav4    2002    180   750
chevy   impala  1985    85    1550
ford    explor  2003    25    9500

Missing pattern

A simple awk program is

{ print }

This program consists of one program line that is an action. Because the pattern is missing, awk selects all lines of input. When used without any arguments the print command displays each selected line in its entirety. This program copies the input to standard output.

$ awk '{ print }' cars
plym    fury    1970    73    2500
chevy   malibu  1999    60    3000
ford    mustang 1965    45    10000
volvo   s80     1998    102   9850
...

Missing action

The next program has a pattern but no explicit action. The slashes indicate that chevy is a regular expression.

/chevy/

In this case awk selects from the input all lines that contain the string chevy. When you do not specify an action, awk assumes that the action is print. The following example copies to standard output all lines from the input that contain the string chevy:

$ awk '/chevy/' cars
chevy   malibu  1999    60      3000
chevy   malibu  2000    50      3500
chevy   impala  1985    85      1550

Single quotation marks

Although neither awk nor shell syntax requires single quotation marks on the command line, it is still a good idea to use them because they can prevent problems. If the awk program you create on the command line includes SPACEs or special shell characters, you must quote them. Always enclosing the program in single quotation marks is the easiest way of making sure that you have quoted any characters that need to be quoted.

Fields

The next example selects all lines from the file (it has no pattern). The braces enclose the action; you must always use braces to delimit the action so that awk can distinguish it from the pattern. This example displays the third field ($3), a SPACE (the output field separator, indicated by the comma), and the first field ($1) of each selected line:

$ awk '{print $3, $1}' cars
1970 plym
1999 chevy
1965 ford
1998 volvo
...

The next example, which includes both a pattern and an action, selects all lines that contain the string chevy and displays the third and first fields from the lines it selects:

$ awk '/chevy/ {print $3, $1}' cars
1999 chevy
2000 chevy
1985 chevy

Next awk selects lines that contain a match for the regular expression h. Because there is no explicit action, awk displays all the lines it selects:

$ awk '/h/' cars
chevy   malibu  1999   60     3000
ford    thundbd 2003   15     10500
chevy   malibu  2000   50     3500
honda   accord  2001   30     6000
chevy   impala  1985   85     1550

~ (matches operator)

The next pattern uses the matches operator (~) to select all lines that contain the letter h in the first field:

$ awk '$1 ~ /h/' cars
chevy   malibu  1999   60     3000
chevy   malibu  2000   50     3500
honda   accord  2001   30     6000
chevy   impala  1985   85     1550

The caret (^) in a regular expression forces a match at the beginning of the line (page 899) or, in this case, the beginning of the first field:

$ awk '$1 ~ /^h/' cars
honda   accord  2001   30     6000

Brackets surround a character-class definition (page 899). In the next example, awk selects lines that have a second field that begins with t or m and displays the third and second fields, a dollar sign, and the fifth field. Because there is no comma between the "$" and the $5, awk does not put a SPACE between them in the output.

$ awk '$2 ~ /^[tm]/ {print $3, $2, "$" $5}' cars
1999 malibu $3000
1965 mustang $10000
2003 thundbd $10500
2000 malibu $3500
2004 taurus $17000

Dollar signs

The next example shows three roles a dollar sign can play in an awk program. A dollar sign followed by a number names a field. Within a regular expression a dollar sign forces a match at the end of a line or field (5$). Within a string a dollar sign represents itself.

$ awk '$3 ~ /5$/ {print $3, $1, "$" $5}' cars
1965 ford $10000
1985 bmw $450
1985 chevy $1550

In the next example, the equal-to relational operator (= =) causes awk to perform a numeric comparison between the third field in each line and the number 1985. The awk command takes the default action, print, on each line where the comparison is true.

$ awk '$3 == 1985' cars
bmw     325i    1985     115     450
chevy   impala  1985     85      1550

The next example finds all cars priced at or less than $3,000:

$ awk '$5 <= 3000' cars
plym    fury    1970    73       2500
chevy   malibu  1999    60       3000
bmw     325i    1985    115      450
toyota  rav4    2002    180      750
chevy   impala  1985    85       1550

Textual comparisons

When you use double quotation marks, awk performs textual comparisons by using the ASCII (or other local) collating sequence as the basis of the comparison. In the following example, awk shows that the strings 450 and 750 fall in the range that lies between the strings 2000 and 9000, which is probably not the intended result:

$ awk '"2000" <= $5 && $5 < "9000"' cars
plym    fury    1970    73      2500
chevy   malibu  1999    60      3000
chevy   malibu  2000    50      3500
bmw     325i    1985    115     450
honda   accord  2001    30      6000
toyota  rav4    2002    180     750

When you need to perform a numeric comparison, do not use quotation marks. The next example gives the intended result. It is the same as the previous example except that it omits the double quotation marks.

$ awk '2000 <= $5 && $5 < 9000' cars
plym    fury    1970    73      2500
chevy   malibu  1999    60      3000
chevy   malibu  2000    50      3500
honda   accord  2001    30      6000

, (range operator)

The range operator (,) selects a group of lines. The first line it selects is the one specified by the pattern before the comma. The last line is the one selected by the pattern after the comma. If no line matches the pattern after the comma, awk selects every line through the end of the input. The next example selects all lines, starting with the line that contains volvo and concluding with the line that contains bmw:

$ awk '/volvo/ , /bmw/' cars
volvo   s80     1998    102    9850
ford    thundbd 2003    15     10500
chevy   malibu  2000    50     3500
bmw     325i    1985    115    450

After the range operator finds its first group of lines, it begins the process again, looking for a line that matches the pattern before the comma. In the following example, awk finds three groups of lines that fall between chevy and ford. Although the fifth line of input contains ford, awk does not select it because at the time it is processing the fifth line, it is searching for chevy.

$ awk '/chevy/ , /ford/' cars
chevy   malibu  1999    60     3000
ford    mustang 1965    45     10000
chevy   malibu  2000    50     3500
bmw     325i    1985    115    450
honda   accord  2001    30     6000
ford    taurus  2004    10     17000
chevy   impala  1985    85     1550
ford    explor  2003    25     9500

f option

When you are writing a longer awk program, it is convenient to put the program in a file and reference the file on the command line. Use the f option followed by the name of the file containing the awk program.

BEGIN

The following awk program, stored in a file named pr_header, has two actions and uses the BEGIN pattern. The awk utility performs the action associated with BEGIN before processing any lines of the data file: It displays a header. The second action, {print}, has no pattern part and displays all the lines from the input.

$ cat pr_header
BEGIN   {print "Make    Model    Year    Miles    Price"}
        {print}

$ awk -f pr_header cars
Make    Model   Year    Miles    Price
plym    fury    1970    73       2500
chevy   malibu  1999    60       3000
ford    mustang 1965    45       10000
volvo   s80     1998    102      9850
...

The next example expands the action associated with the BEGIN pattern. In the previous and following examples, the whitespace in the headers is composed of single TABs, so that the titles line up with the columns of data.

$ cat pr_header2
BEGIN   {
print "Make    Model    Year    Miles    Price"
print "----------------------------------------"
}
        {print}
$ awk -f pr_header2 cars
Make    Model   Year    Miles    Price
----------------------------------------
plym    fury    1970    73       2500
chevy   malibu  1999    60       3000
ford    mustang 1965    45       10000
volvo   s80     1998    102      9850
...

length function

When you call the length function without an argument, it returns the number of characters in the current line, including field separators. The $0 variable always contains the value of the current line. In the next example, awk prepends the line length to each line and then a pipe sends the output from awk to sort (the n option specifies a numeric sort; page 837) so that the lines of the cars file appear in order of length:

$ awk '{print length, $0}' cars | sort -n
21 bmw  325i    1985    115     450
22 plym fury    1970    73      2500
23 volvo        s80     1998    102    9850
24 ford explor  2003    25      9500
24 toyota       rav4    2002    180    750
25 chevy        impala  1985    85     1550
25 chevy        malibu  1999    60     3000
25 chevy        malibu  2000    50     3500
25 ford taurus  2004    10      17000
25 honda        accord  2001    30     6000
26 ford mustang 1965    45      10000
26 ford thundbd 2003    15      10500

The formatting of this report depends on TABs for horizontal alignment. The three extra characters at the beginning of each line throw off the format of several lines. A remedy for this situation is covered shortly.

NR (record number)

The NR variable contains the record (line) number of the current line. The following pattern selects all lines that contain more than 24 characters. The action displays the line number of each of the selected lines.

$ awk 'length > 24 {print NR}' cars
2
3
5
6
8
9
11

You can combine the range operator (,) and the NR variable to display a group of lines of a file based on their line numbers. The next example displays lines 2 through 4:

$ awk 'NR == 2 , NR == 4' cars
chevy   malibu  1999    60     3000
ford    mustang 1965    45     10000
volvo   s80     1998    102    9850

END

The END pattern works in a manner similar to the BEGIN pattern, except that awk takes the actions associated with it after processing the last line of input. The following report displays information only after it has processed all the input. The NR variable retains its value after awk finishes processing the data file, so that an action associated with an END pattern can use it:

$ awk 'END {print NR, "cars for sale." }' cars
12 cars for sale.

The next example uses if control structures to expand the abbreviations used in some of the first fields. As long as awk does not change a record, it leaves the entire recordincluding separatorsintact. Once it makes a change to a record, awk changes all separators in that record to the value of the output field separator. The default output field separator is a SPACE.

$ cat separ_demo
        {
        if ($1 ~ /ply/)  $1 = "plymouth"
        if ($1 ~ /chev/) $1 = "chevrolet"
        print
        }

$ awk -f separ_demo cars
plymouth fury 1970 73 2500
chevrolet malibu 1999 60 3000
ford    mustang 1965    45    10000
volvo   s80     1998    102   9850
ford    thundbd 2003    15    10500
chevrolet malibu 2000 50 3500
bmw     325i    1985    115   450
honda   accord  2001    30    6000
ford    taurus  2004    10    17000
toyota  rav4    2002    180   750
chevrolet impala 1985 85 1550
ford    explor  2003    25    9500

Stand-alone script

Instead of calling awk from the command line with the f option and the name of the program you want to run, you can write a script that calls awk with the commands you want to run. The next example is a stand-alone script that runs the same program as the previous example. The #!/usr/bin/awkf command (page 267) runs the awk utility directly. You need both read and execute permission to the file holding the script (page 265).

$ chmod u+rx separ_demo2
$ cat separ_demo2
#!/usr/bin/awk -f
        {
        if ($1 ~ /ply/)  $1 = "plymouth"
        if ($1 ~ /chev/) $1 = "chevrolet"
        print
        }
$ separ_demo2 cars
plymouth fury 1970 73 2500
chevrolet malibu 1999 60 3000
ford    mustang 1965    45     10000
...

OFS variable

You can change the value of the output field separator by assigning a value to the OFS variable. The following example assigns a TAB character to OFS, using the backslash escape sequence \t. This fix improves the appearance of the report but does not line up the columns properly.

$ cat ofs_demo
BEGIN   {OFS = "\t"}
        {
        if ($1 ~ /ply/)  $1 = "plymouth"
        if ($1 ~ /chev/) $1 = "chevrolet"
        print
        }

$ awk -f ofs_demo cars
plymouth        fury    1970    73     2500
chevrolet       malibu  1999    60     3000
ford    mustang 1965    45      10000
volvo   s80     1998    102     9850
ford    thundbd 2003    15      10500
chevrolet       malibu  2000    50     3500
bmw     325i    1985    115     450
honda   accord  2001    30      6000
ford    taurus  2004    10      17000
toyota  rav4    2002    180     750
chevrolet       impala  1985    85     1550
ford    explor  2003    25      9500

printf

You can use printf (page 615) to refine the output format. The following example uses a backslash at the end of several program lines to quote the following NEWLINE. You can use this technique to continue a long line over one or more lines without affecting the outcome of the program.

$ cat printf_demo
BEGIN   {
    print "                                  Miles"
    print "Make       Model       Year       (000)      Price"
    print \
    "--------------------------------------------------"
    }
    {
    if ($1 ~ /ply/)  $1 = "plymouth"
    if ($1 ~ /chev/) $1 = "chevrolet"
    printf "%-10s %-8s    %2d    %5d     $ %8.2f\n",\
       $1, $2, $3, $4, $5
    }

$ awk -f printf_demo cars
                                Miles
Make       Model       Year     (000)       Price
--------------------------------------------------
plymouth   fury        1970     73      $  2500.00
chevrolet  malibu      1999     60      $  3000.00
ford       mustang     1965     45      $ 10000.00
volvo      s80         1998    102      $  9850.00
ford       thundbd     2003     15      $ 10500.00
chevrolet  malibu      2000     50      $  3500.00
bmw        325i        1985    115      $   450.00
honda      accord      2001     30      $  6000.00
ford       taurus      2004     10      $ 17000.00
toyota     rav4        2002    180      $   750.00
chevrolet  impala      1985     85      $  1550.00
ford       explor      2003     25      $  9500.00

Redirecting output

The next example creates two files: one with the lines that contain chevy and one with the lines that contain ford:

$ cat redirect_out
/chevy/     {print > "chevfile"}
/ford/      {print > "fordfile"}
END         {print "done."}

$ awk -f redirect_out cars
done.
$ cat chevfile
chevy   malibu  1999    60     3000
chevy   malibu  2000    50     3500
chevy   impala  1985    85     1550

The summary program produces a summary report on all cars and newer cars. Although they are not required, the initializations at the beginning of the program represent good programming practice; awk automatically declares and initializes variables as you use them. After reading all the input data, awk computes and displays averages.

$ cat summary
BEGIN   {
        yearsum = 0 ; costsum = 0
        newcostsum = 0 ; newcount = 0
        }
        {
        yearsum += $3
        costsum += $5
        }
$3 > 2000 {newcostsum += $5 ; newcount ++}
END     {
        printf "Average age of cars is %4.1f years\n",\
            2006 - (yearsum/NR)
        printf "Average cost of cars is $%7.2f\n",\
            costsum/NR
            printf "Average cost of newer cars is $%7.2f\n",\
                newcostsum/newcount
        }
$ awk -f summary cars
Average age of cars is 13.1 years
Average cost of cars is $6216.67
Average cost of newer cars is $8750.00

The following awk command shows the format of a line from the passwd database that the next example uses:

$ nidump passwd / | awk '/mark/ {print}'
mark:********:107:100::0:0:ext 112:/Users/mark:/bin/tcsh

In this example the nidump utility (page 796) sends the passwd database to standard output.

The next example demonstrates a technique for finding the largest number in a field. Because it works with the passwd database, which delimits fields with colons (:), the example changes the input field separator (FS) before reading any data. It reads the passwd database and determines the next available user ID number (field 3). The numbers do not have to be in order in the passwd database for this program to work.

The pattern ($3 > saveit) causes awk to select records that contain a user ID number greater than any previous user ID number that it has processed. Each time it selects a record, awk assigns the value of the new user ID number to the saveit variable. Then awk uses the new value of saveit to test the user IDs of all subsequent records. Finally awk adds 1 to the value of saveit and displays the result.

$ cat find_uid
BEGIN          {FS = ":"
               saveit = 0}
$3 > saveit    {saveit = $3}
END            {print "Next available UID is " saveit + 1}

$ nidump passwd / | awk -f find_uid
Next available UID is 1029

The next example produces another report based on the cars file. This report uses nested if...else control structures to substitute values based on the contents of the price field. The program has no pattern part; it processes every record.

$ cat price_range
    {
    if            ($5 <= 5000)                $5 = "inexpensive"
       else if    (5000 < $5 &&  $5 < 10000)  $5 = "please ask"
       else if    (10000 <= $5)               $5 = "expensive"
    #
    printf "%-10s %-8s    %2d    %5d    %-12s\n",\
    $1, $2, $3, $4, $5
    }
$ awk -f price_range cars
plym       fury        1970       73     inexpensive
chevy      malibu      1999       60     inexpensive
ford       mustang     1965       45     expensive
volvo      s80         1998      102     please ask
ford       thundbd     2003       15     expensive
chevy      malibu      2000       50     inexpensive
bmw        325i        1985      115     inexpensive
honda      accord      2001       30     please ask
ford       taurus      2004       10     expensive
toyota     rav4        2002      180     inexpensive
chevy      impala      1985       85     inexpensive
ford       explor      2003       25     please ask

Associative arrays

Next the manuf associative array uses the contents of the first field of each record in the cars file as an index. The array is composed of the elements manuf[plym], manuf[chevy], manuf[ford], and so on. Each new element is initialized to 0 (zero) as it is created. The C language operator ++ increments the variable that it follows.

The action following the END pattern is the special for structure that loops through the elements of an associative array. A pipe sends the output through sort to produce an alphabetical list of cars and the quantities in stock. Because it is a shell script and not an awk program file, you must have both read and execute permission to the manuf file to execute it as a command. Depending on how the PATH variable (page 285) is set, you may have to execute the script as ./manuf.

$ cat manuf
awk '   {manuf[$1]++}
END     {for (name in manuf) print name, manuf[name]}
' cars |
sort

$ manuf
bmw 1
chevy 3
ford 4
honda 1
plym 1
toyota 1
volvo 1

The next program, named manuf.sh, is a more general shell script that includes some error checking. This script lists and counts the contents of a column in a file, with both the column number and the name of the file being specified on the command line.

The first action (the one that starts with {count) uses the shell variable $1 in the middle of the awk program to specify an array index. Because of the way the single quotation marks are paired, the $1 that appears to be within single quotation marks is actually not quoted: The two quoted strings in the awk program surround, but do not include, the $1. Because the $1 is not quoted, and because this is a shell script, the shell substitutes the value of the first command line argument in place of $1 (page 565). As a result the $1 is interpreted before the awk command is invoked. The leading dollar sign (the one before the first single quotation mark on that line) causes awk to interpret what the shell substitutes as a field number.

$ cat manuf.sh
if [ $# != 2 ]
    then
        echo "Usage: manuf.sh field file"
        exit 1
fi
awk < $2 '
       {count[$'$1']++}
END    {for (item in count) printf "%-20s%-20s\n",\
           item, count[item]}' |
sort

$ manuf.sh
Usage: manuf.sh field file

$ manuf.sh 1 cars
bmw                 1
chevy               3
ford                4
honda               1
plym                1
toyota              1
volvo               1

$ manuf.sh 3 cars
1965                1
1970                1
1985                2
1998                1
1999                1
2000                1
2001                1
2002                1
2003                2
2004                1

A way around the tricky use of quotation marks that allow parameter expansion within the awk program is to use the v option on the command line to pass the field number to awk as a variable. This change makes it easier for someone else to read and debug the script. You call the manuf2.sh script the same way you call manuf.sh:

$ cat manuf2.sh
if [ $# != 2 ]
        then
                echo "Usage: manuf.sh field file"
                gexit 1
fi
awk -v "field=$1" < $2 '
                {count[$field]++}
END             {for (item in count) printf "%-20s%-20s\n",\
                        item, count[item]}' |
sort

The word_usage script displays a word usage list for a file you specify on the command line. The TR utility (page 879) lists the words from standard input, one to a line. The sort utility orders the file, with the most frequently used words first. This script sorts groups of words that are used the same number of times in alphabetical order.

$ cat word_usage
tr -cs 'a-zA-Z' '[\n*]' < $1 |
awk     '
        {count[$1]++}
END     {for (item in count) printf "%-15s%3s\n", item, count[item]}' |
sort +1nr +0f -1

$ word_usage textfile
the             42
file            29
fsck            27
system          22
you             22
to              21
it              17
SIZE            14
and             13
MODE            13
...

Following is a similar program in a different format. The style mimics that of a C program and may be easier to read and work with for more complex awk programs:

$ cat word_count
tr -cs 'a-zA-Z' '[\n*]' < $1 |
awk '   {
        count[$1]++
}
END     {
        for (item in count)
            {
            if (count[item] > 4)
                {
                printf "%-15s%3s\n", item, count[item]
                }
        }
} ' |
sort +1nr +0f -1

The tail utility displays the last ten lines of output, illustrating that words occurring fewer than five times are not listed:

$ word_count textfile | tail
directories      5
if               5
information      5
INODE            5
more             5
no               5
on               5
response         5
this             5
will             5

The next example shows one way to put a date on a report. The first line of input to the awk program comes from date. The program reads this line as record number 1 (NR == 1), processes it accordingly, and processes all subsequent lines with the action associated with the next pattern (NR > 1).

$ cat report
if (test $# = 0) then
   echo "You must supply a filename."
   exit 1
fi
(date; cat $1) |
awk '
NR == 1    {print "Report for", $1, $2, $3 ", " $6}
NR >  1    {print $5 "\t" $1}'

$ report cars
Report for Mon Jan 31, 2005
2500    plym
3000    chevy
10000   ford
9850    volvo
10500   ford
3500    chevy
450     bmw
6000    honda
17000   ford
750     toyota
1550     chevy
9500    ford

The next example sums each of the columns in a file you specify on the command line; it takes its input from the numbers file. The program performs error checking, reporting on and discarding rows that contain nonnumeric entries. It uses the next command (with the comment skip bad records) to skip the rest of the commands for the current record if the record contains a nonnumeric entry. At the end of the program, awk displays a grand total for the file.

$ cat numbers
10      20       30.3    40.5
20      30       45.7    66.1
30      xyz      50      70
40      75       107.2   55.6
50      20       30.3    40.5
60      30       45.0    66.1
70      1134.7   50      70
80      75       107.2   55.6
90      176      30.3    40.5
100     1027.45  45.7    66.1
110     123      50      57a.5
120     75       107.2   55.6
$ cat tally
awk '  BEGIN   {
               ORS = ""
               }
NR == 1 {                                        # first record only
    nfields = NF                                 # set nfields to number of
    }                                            # fields in the record (NF)
    {
    if ($0 ~ /[^0-9. \t]/)                       # check each record to see if it contains
        {                                        # any characters that are not numbers,
        print "\nRecord " NR " skipped:\n\t"     # periods, spaces, or TABs
        print $0 "\n"
        next                                     # skip bad records
        }
    else
        {
        for (count = 1; count <= nfields; count++)# for good records loop through fields
            {
            printf "%10.2f", $count > "tally.out"
            sum[count] += $count
            gtotal += $count
            }
        print "\n" > "tally.out"
        }
    }
END     {                                        # after processing last record
    for (count = 1; count <= nfields; count++)   # print summary
        {
        print "  -------" > "tally.out"
        }
    print "\n" > "tally.out"
    for (count = 1; count <= nfields; count++)
        {
        printf "%10.2f", sum[count] > "tally.out"
        }
    print "\n\n        Grand Total" gtotal "\n" > "tally.out"
} ' < numbers

                   $ tally
                   Record  3 skipped:
                            30      xyz     50      70

                   Record 6 skipped:
                           60      30       45.0    66.1

                   Record 11 skipped:
                           110     123      50      57a.5
$ cat tally.out
     10.00     20.00     30.30     40.50
     20.00     30.00     45.70     66.10
     40.00     75.00    107.20     55.60
     50.00     20.00     30.30     40.50
     70.00   1134.70     50.00     70.00
     80.00     75.00    107.20     55.60
     90.00    176.00     30.30     40.50
    100.00   1027.45     45.70     66.10
    120.00     75.00    107.20     55.60
   -------   -------   -------   -------
    580.00   2633.15    553.90    490.50

        Grand Total 4257.55

The next example shows a complete interactive shell script that uses awk to generate a report on the cars file based on price ranges:

$ cat list_cars
trap 'rm -f $$.tem > /dev/null;echo $0 aborted.;exit 1' 1 2 15
echo -n "Price range (for example, 5000 7500):"
read lowrange hirange

echo '
                                 Miles
Make        Model       Year     (000)       Price
--------------------------------------------------' > $$.tem
awk < cars '
$5 >= '$lowrange' && $5 <= '$hirange' {
        if ($1 ~ /ply/) $1 = "plymouth"
        if ($1 ~ /chev/) $1 = "chevrolet"
        printf "%-10s %-8s    %2d    %5d    $ %8.2f\n", $1, $2, $3, $4,
$5
        }' | sort -n +5 >> $$.tem
cat $$.tem
rm $$.tem

$ list_cars
Price range (for example, 5000 7500):3000 8000

                               Miles
Make      Model      Year      (000)        Price
-------------------------------------------------
chevrolet  malibu    1999         60    $ 3000.00
chevrolet  malibu    2000         50    $ 3500.00
honda      accord    2001         30    $ 6000.00
$ list_cars
Price range (for example, 5000 7500):0 2000

                               Miles
Make       Model      Year     (000)        Price
-------------------------------------------------
bmw        325i       1985       115    $  450.00
toyota     rav4       2002       180    $  750.00
chevrolet  impala     1985        85    $ 1550.00

$ list_cars
Price range (for example, 5000 7500):15000 100000

                               Miles
Make       Model      Year     (000)         Price
--------------------------------------------------
ford       taurus     2004        10    $ 17000.00

Optional: Advanced awk Programming

This section discusses two advanced features of awk. It covers how to control input using the getline statement and how to use the system statement to execute commands within an awk program.

getline: CONTROLLING INPUT

The getline statement gives you more control over the data awk reads than other methods of input do. When you give a variable name as an argument to getline, it reads data into that variable. The BEGIN block of the g1 program uses getline to read one line into the variable aa from standard input:

$ cat g1
BEGIN   {
        getline aa
        print aa
        }
$ echo aaaa | awk -f g1
aaaa

The alpha file is used in the next few examples:

$ cat alpha
aaaaaaaaa
bbbbbbbbb
ccccccccc
ddddddddd

Even when g1 is given more than one line of input, it processes only the first line:

$ awk -f g1 < alpha
aaaaaaaaa

When getline is not given an argument, it reads into $0 and modifies the field variables ($1, $2, ...):

$ awk 'BEGIN {getline;print $1}' < alpha
aaaaaaaaa

The g2 program uses a while loop in the BEGIN block to loop over the lines in standard input. The getline statement reads each line into holdme and print outputs each value of holdme.

$ cat g2
BEGIN   {
        while (getline holdme)
            print holdme
        }
$ awk -f g2 < alpha
aaaaaaaaa
bbbbbbbbb
ccccccccc
ddddddddd

The g3 program demonstrates that awk automatically reads each line of input into $0 when it has statements in its body (and not just a BEGIN block). This program outputs the record number (NR), the string $0:, and the value of $0 (the current record) for each line of input.

$ cat g3
        {print NR, "$0:", $0}

$ awk -f g3 < alpha
1 $0: aaaaaaaaa
2 $0: bbbbbbbbb
3 $0: ccccccccc
4 $0: ddddddddd

Next g4 demonstrates that getline works independently of awk's automatic reads and $0. When getline reads into a variable, it does not modify $0 nor does it modify any of the fields in the current record ($1, $2, . . .). The first statement in g4 is the same as the statement in g3 and outputs the line that awk has automatically read. The getline statement reads the next line of input into the variable named aa. The third statement outputs the record number, the string aa:, and the value of aa. The output from g4 shows that getline processes records independently of awk's automatic reads.

$ cat g4
        {
        print NR, "$0:", $0
        getline aa
        print NR, "aa:", aa
        }
$ awk -f g4 < alpha
1 $0: aaaaaaaaa
2 aa: bbbbbbbbb
3 $0: ccccccccc
4 aa: ddddddddd

The g5 program outputs each line of input except that it skips lines that begin with the letter b. The first print statement outputs each line that awk reads automatically. Next the /^b/ pattern selects all lines that begin with b for special processing. The action uses getline to read the next line of input into the variable hold, outputs the string skip this line: followed by the value of hold, and outputs the value of $1. The $1 holds the value of the first field of the record that awk read automatically, not the record read by getline. The final statement displays a string and the value of NR, the current record number. Even though getline does not change $0 when it reads into a variable, awk increments NR.

$ cat g5
        # print all lines except those read with getline
        {print "line #", NR, $0}

# if line begins with "b" process it specially
/^b/    {
        # use getline to read the next line into variable named hold
        getline hold

        # print value of hold
        print "skip this line:", hold

        # $0 is not affected when getline reads into a variable
        # $1 still holds previous value
        print "previous line began with:", $1
        }

        {
        print ">>>> finished processing line #", NR
        print ""
        }

$ awk -f g5 < alpha
line # 1 aaaaaaaaa
>>>> finished processing line # 1

line # 2 bbbbbbbbb
skip this line: ccccccccc
previous line began with: bbbbbbbbb
>>>> finished processing line # 3

line # 4 ddddddddd
>>>> finished processing line # 4

system: RUNNING EXTERNAL COMMANDS

The system statement runs OS X commands, such as utilities, from within an awk program. The following example uses system to get the date and then to copy files.

$ cat backup
BEGIN   {
       system("date +%m%d > /tmp/awkfile");
       getline date < "/tmp/awkfile";
       if (system("mkdir " date))
           {
           print "Error trying to create directory", date;
           exit;
           }
       getline dummy;
       close("/tmp/awkfile")
       }
!/^d/  {
       system("cp " $9 " " date "/" $9);
       }
$ ls -l | awk -f backup

The return code of system is the exit status (page 564) of the command that it ran. Usually a zero indicates success and a nonzero value indicates failure.


Error Messages

The following examples show some of the more common causes of awk error messages (and nonmessages). These examples are run under bash. When you use awk with tcsh, the error messages from the shell will be different.

The first example leaves the single quotation marks off the command line, so the shell interprets $3 and $1 as shell variables. Also, because there are no single quotation marks, the shell passes awk four arguments instead of two.

$ awk {print $3, $1} cars
awk: syntax error at source line 1
 context is
         >>>  <<<
awk: illegal statement at source line 1
        missing }

The next command line includes a typo (prinnt) that awk does not catch. Instead of issuing an error message, awk simply does not do anything useful.

$ awk '$3 >= 83 {prinnt $1}' cars

The next example has no braces around the action:

$ awk '/chevy/ print $3, $1' cars
awk: syntax error at source line 1
 context is
        /chevy/ >>>  print <<<  $3, $1
awk: bailing out at source line 1

There is no problem with the next example; awk did just what you asked it to do (none of the lines in the file contains a z).

$ awk '/z/' cars

The next example shows an improper action for which awk does not issue an error message:

$ awk '{$3 " made by " $1}' cars

The following example does not display the heading because there is no backslash after the print command in the BEGIN block. The backslash is needed to quote the following NEWLINE so that the line can be continued. Without it, awk sees two separate statements; the second is invalid.

$ cat print_cars
BEGIN   {print
"Model  Year    Price"}
/chevy/ {printf "%5s\t%4d\t%5d\n", $2, $3, $5}
$ awk -f print_cars cars
awk: illegal statement
 source line number 2

You must use double quotation marks, not single ones, to delimit strings.

$ cat print_cars2
BEGIN {OFS='\t'}
$3 ~ /5$/ {print $3, $1, "$" $5}

$ awk -f print_cars2 cars
awk: syntax error at source line 1 source file print_cars2
 context is
        BEGIN >>> {OFS=' <<<
awk: illegal statement at source line 1 source file print_cars2


Chapter Summary

The awk utility is a pattern-scanning and processing language that searches one or more files to see whether they contain records (usually lines) that match specified patterns. It processes lines by performing actions, such as writing the record to standard output or incrementing a counter, each time it finds a match.

An awk program consists of one or more lines containing a pattern and/or action in the following format:

pattern { action}

The pattern selects lines from the input. The awk utility performs the action on all lines that the pattern selects. If a program line does not contain a pattern, awk selects all lines in the input. If a program line does not contain an action, awk copies the selected lines to standard output.

An awk program can use variables, functions, arithmetic operators, associative arrays, control statements, and C's printf statement. Advanced awk programming takes advantage of getline statements to fine-tune input, and system statements to run OS X commands from within an awk program.


Exercises

1. | Write an awk program that numbers each line in a file and sends its output to standard output.
2. | Write an awk program that displays the number of characters in the first field followed by the first field and sends its output to standard output.
3. | Write an awk program that uses the cars file (page 618), displays all cars priced at more than $5000, and sends its output to standard output.
4. | Use awk to determine how many lines in /etc/services contain the string Unassigned. Verify your answer using grep.


Advanced Exercises

5. | Use awk to build a program that displays a process list as a tree, showing each process indented under its parent process. (Hint: You may want to use a recursive function in your solution.)
6. | Write an shell script named listproc that uses awk and displays the PID numbers of processes whose names match the argument given to the script.


Chapter 15. The sed Editor

IN THIS CHAPTER

Syntax | 642
Arguments | 642
Options | 642
Editor Basics | 642
Addresses | 643
Instructions | 643
Control Structures | 645
The Pattern Space and the Hold Space | 645
Examples | 646

The sed (stream editor) utility is a batch (noninteractive) editor. It transforms an input stream that can come from a file or standard input. It is frequently used as a filter in a pipe. Because it makes only one pass through its input, sed is more efficient than an interactive editor such as ed.


Syntax

A sed command line has the following syntax:

sed [-n] program [file-list]
sed [-n] -f program-file [file-list]

The sed utility takes its input from files you specify on the command line or from standard input. Unless you direct output from sed elsewhere, it goes to standard output.


Arguments

The program is a sed program that is included on the command line. This format allows you to write simple, short sed programs without creating a separate program-file. The program-file is the pathname of a file containing a sed program (see "Editor Basics"). The file-list contains pathnames of the ordinary files that sed processes. These are the input files. When you do not specify a file, sed takes its input from standard input.


Options

E | (extended) Uses extended regular expressions (page 903) in place of basic ones.
f | program-file
 | Causes sed to read its program from the file named program-file instead of from the command line. You can use this option more than once on the command line.
i | [suffix]
 | Edits files in place. Without this option sed sends its output to standard output. With this option sed replaces the file it is processing with its output. When you specify a suffix, sed makes a backup of the original file. This backup has the original filename with suffix appended. You must include a period in suffix if you want a period to appear between the original filename and suffix.
n | Causes sed not to copy lines to standard output except as specified by the Print (p) instruction or flag.


Editor Basics

A sed program consists of one or more lines with the following syntax:

[address[,address]]instruction [argument-list]

The addresses are optional. If you omit the address, sed processes all lines from the input. The instruction is the editing instruction that modifies the text. The addresses select the line(s) that the instruction part of the command operates on. The number and kinds of arguments in the argument-list depend on the instruction. If you want to put several sed commands on one line you can separate the commands with semicolons (;).

The sed utility processes input as follows:

1. | Reads one line of input from file-list or standard input.
2. | Reads the first instruction from the program or program-file. If the address(es) select the input line, acts on the input line as the instruction specifies.
3. | Reads the next instruction from the program or program-file. If the address(es) select the input line, acts on the input line (possibly modified by the previous instruction) as the instruction specifies.
4. | Repeats step 3 until it has executed all instructions in the program or program-file.
5. | Starts over again with step 1 if there is another line of input; otherwise, it is finished.

Addresses

A line number is an address that selects a line. As a special case, the line number $ represents the last line of input.

A regular expression (refer to Appendix A) is an address that selects those lines containing a string that the regular expression matches. Although slashes are often used to delimit these regular expressions, sed permits you to use any character other than a backslash or NEWLINE for this purpose.

Except as noted, zero, one, or two addresses (either line numbers or regular expressions) can precede an instruction. If you do not use an address, sed selects all lines, causing the instruction to act on every line of input. Providing one address causes the instruction to act on each input line that the address selects. Providing two addresses causes the instruction to act on groups of lines. The first address selects the first line in the first group. The second address selects the next subsequent line that it matches; this line is the last line in the first group. If no match for the second address is found, the second address points to the end of the file. After selecting the last line in a group, sed starts the selection process over again, looking for the next line that the first address matches. This line is the first line in the next group. The sed utility continues this process until it has finished going through the entire file.

Instructions

d

(delete) The Delete instruction causes sed not to write out the lines it selects and not to finish processing the lines. After sed executes a Delete instruction, it reads the next input line and begins over again with the first instruction from the program or program-file.

n

(next) The Next instruction writes out the currently selected line if appropriate, reads the next input line, and starts processing the new line with the next instruction from the program or program-file.

a

(append) The Append instruction appends one or more lines to the currently selected line. If you precede an Append instruction with two addresses, it appends to each line that is selected by the addresses; earlier versions of sed did not accept Append instructions with two addresses. If you do not precede an Append instruction with an address, it appends to each input line. An Append instruction has the following format:

[address[,address]] a\
text\
text\
...
text

You must end each line of appended text, except the last, with a backslash (the backslash quotes the following NEWLINE). The appended text concludes with a line that does not end with a backslash. The sed utility always writes out appended text, regardless of whether you use a -n flag on the command line. It even writes out the text if you delete the line to which you are appending the text.

i

(insert) The Insert instruction is identical to the Append instruction except that it places the new text before the selected line.

c

(change) The Change instruction is similar to Append and Insert except that it changes the selected lines so that they contain the new text. When you specify an address range, Change replaces the entire range of lines with a single occurrence of the new text.

s

(substitute) The Substitute instruction in sed is similar to that in vim (page 172). It has the following format:

[address[,address]] s/pattern/replacement-string/[g][p][w file]

The pattern is a regular expression (refer to Appendix A) that is delimited by any character other than a SPACE or NEWLINE; traditionally a slash (/). The replacement-string starts immediately following the second delimiter and must be terminated by the same delimiter. The final (third) delimiter is required. The replacement-string can contain an ampersand (&), which sed replaces with the matched pattern. Unless you use the g flag, the Substitute instruction replaces only the first occurrence of the pattern on each selected line.

The g (global) flag causes the Substitute instruction to replace all nonoverlapping occurrences of the pattern on the selected lines.

The p (print) flag causes sed to send all lines on which it makes substitutions to standard output. This flag overrides the -n option on the command line.

The w (write) flag is similar to the p flag but it sends its output to the file specified by file. A single SPACE and the name of the output file must follow a w flag.

p

(print) The Print instruction writes the selected lines to standard output, writing the lines immediately, and does not reflect the effects of subsequent instructions. This instruction overrides the -n option on the command line. (The -n option prevents sed from copying lines to standard output.)

w file

(write) This instruction is similar to the Print instruction except that it sends output to the file specified by file. A single SPACE and the name of the output file must follow a Write instruction.

r file

(read) The Read instruction reads the contents of the specified file and appends it to the selected line. A single SPACE and the name of the input file must follow a Read instruction.

q

(quit) The Quit instruction causes sed to terminate immediately.

Control Structures

!

(NOT) Causes sed to apply the following instruction, located on the same line, to each of the lines not selected by the address portion of the instruction. For example, 3!d deletes all lines except line 3 and $!p displays all lines except the last.

{}

(group instructions) When you enclose a group of instructions within a pair of braces, a single address (or address pair) selects the lines on which the group of instructions operates. Use semicolons (;) to separate multiple commands appearing on a single line.

Branch instructions

The GNU sed info page lists the branch instructions as "Commands for sed gurus" and suggests that if you need them you might be better off writing your program in awk or Perl.

: label

Identifies a location within a sed program. The label is useful as a target for the b and t branch instructions.

b [label]

Unconditionally transfers control (branches) to label. Without label, skips the rest of the instructions for the current line of input and reads the next line of input (page 643).

t [label]

Transfers control (branches) to label only if a Substitute instruction has been successful since the most recent line of input was read (conditional branch). Without label, skips the rest of the instructions for the current line of input and reads the next line of input (page 643).

The Pattern Space and the Hold Space

The sed utility has two buffers. The commands reviewed up to this point work with the Pattern space, which initially holds the line of input that sed just read. The Hold space can hold data while you manipulate data in the Pattern space; it is a temporary buffer. Until you place data in the Hold space it is empty. This section discusses commands that move data between the Pattern space and the Hold space.

g

Copies the contents of the Hold space to the Pattern space. The original contents of the Pattern space is lost.

G

Appends a NEWLINE and the contents of the Hold space to the Pattern space.

h

Copies the contents of the Pattern space to the Hold space. The original contents of the Hold space is lost.

H

Appends a NEWLINE and the contents of the Pattern space to the Hold space.

x

Exchanges the contents of the Pattern space and the Hold space.


Examples

new data file

The following examples use the input file new:

$ cat new
Line one.
The second line.
The third.
This is line four.
Five.
This is the sixth sentence.
This is line seven.
Eighth and last.

Unless you instruct it not to, sed sends all linesselected or notto standard output. When you use the -n option on the command line, sed sends only certain lines, such as those selected by a Print (p) instruction, to standard output.

The following command line displays all the lines in the new file that contain the word line (all lowercase). In addition, because there is no -n option, sed displays all the lines of input. As a result the lines that contain the word line are displayed twice.

$ sed '/line/ p' new
Line one.
The second line.
The second line.
The third.
This is line four.
This is line four.
Five.
This is the sixth sentence.
This is line seven.
This is line seven.
Eighth and last.

The command uses the address /line/, a regular expression that is a simple string. The sed utility selects each of the lines that contains a match for that pattern. The Print (p) instruction displays each of the selected lines.

The following command uses the -n option so that sed displays only the selected lines:

$ sed -n '/line/ p' new
The second line.
This is line four.
This is line seven.

Next sed displays part of a file based on line numbers. The Print instruction selects and displays lines 3 through 6.

$ sed -n '3,6 p' new
The third.
This is line four.
Five.
This is the sixth sentence.

The next command line uses the Quit instruction to cause sed to display only the beginning of a file. In this case sed displays the first five lines of new just as a head -5 new command would.

$ sed '5 q' new
Line one.
The second line.
The third.
This is line four.
Five.

program-file

When you need to give sed more complex or lengthy instructions, you can use a program-file. The print3_6 program performs the same function as the command line in a previous example. The -f option tells sed that it should read its program from the file named on the command line.

$ cat print3_6
3,6 p

$ sed -n -f print3_6 new
The third.
This is line four.
Five.
This is the sixth sentence.

Append

The next program selects line 2 and uses an Append instruction to append a NEWLINE and the text AFTER. to the selected line. Because the command line does not include the -n option, sed copies all the lines from the input file new.

$ cat append_demo
2 a\
AFTER.

$ sed -f append_demo new
Line one.
The second line.
AFTER.
The third.
This is line four.
Five.
This is the sixth sentence.
This is line seven.
Eighth and last.

Insert

The insert_demo program selects all the lines containing the string This and inserts a NEWLINE and the text BEFORE. before the selected lines.

$ cat insert_demo
/This/ i\
BEFORE.

$ sed -f insert_demo new
Line one.
The second line.
The third.
BEFORE.
This is line four.
Five.
BEFORE.
This is the sixth sentence.
BEFORE.
This is line seven.
Eighth and last.

Change

The next example demonstrates a Change instruction with an address range. When you specify a range of lines for a Change instruction, it does not change each line within the range but rather changes the block of lines to a single occurrence of the new text.

$ cat change_demo
2,4 c\
SED WILL INSERT THESE\
THREE LINES IN PLACE\
OF THE SELECTED LINES.

$ sed -f change_demo new
Line one.
SED WILL INSERT THESE
THREE LINES IN PLACE
OF THE SELECTED LINES.
Five.
This is the sixth sentence.
This is line seven.
Eighth and last.

Substitute

The next example demonstrates a Substitute instruction. The sed utility selects all lines because the instruction has no address. On each line subs_demo replaces the first occurrence of line with sentence. The p flag displays each line where a substitution occurs. The command line calls sed with the -n option, so sed displays only the lines that the program explicitly requests it to display.

$ cat subs_demo
s/line/sentence/p

$ sed -n -f subs_demo new
The second sentence.
This is sentence four.
This is sentence seven.

The next example is similar to the preceding one except that a w flag and filename (temp) at the end of the Substitute instruction cause sed to create the file named temp. The command line does not include the -n option, so it displays all lines in addition to writing the changed lines to temp. The cat utility displays the contents of the file temp. The word Line (starting with an uppercase L) is not changed.

$ cat write_demo1
s/line/sentence/w temp

$ sed -f write_demo1 new
Line one.
The second sentence.
The third.
This is sentence four.
Five.
This is the sixth sentence.
This is sentence seven.
Eighth and last.

$ cat temp
The second sentence.
This is sentence four.
This is sentence seven.

The following bash script changes all occurrences of REPORT to report, FILE to file, and PROCESS to process in a group of files. Because it is a shell script and not a sed program file, you must have read and execute permission to the sub file to execute it as a command (page 265). The for structure (page 538) loops through the list of files supplied on the command line. As it processes each file, the script displays each filename before processing the file with sed. This program uses multiline embedded sed commands. Because the NEWLINEs between the commands are quoted (placed between single quotation marks), sed accepts multiple commands on a single, extended command line (within a shell script). Each Substitute instruction includes a g (global) flag to take care of the case where a string occurs more than one time on a line.

$ cat sub
for file
do
        echo $file
        mv $file $$.subhld
        sed 's/REPORT/report/g
             s/FILE/file/g
             s/PROCESS/process/g' $$.subhld > $file
done
rm $$.subhld

$ sub file1 file2 file3
file1
file2
file3

In the next example, a Write instruction copies part of a file to another file (temp2). The line numbers 2 and 4, separated by a comma, select the range of lines sed is to copy. This program does not alter the lines.

$ cat write_demo2
2,4 w temp2

$ sed -n -f write_demo2 new

$ cat temp2
The second line.
The third.
This is line four.

The program write_demo3 is very similar to write_demo2 but precedes the Write instruction with the NOT operator (!), causing sed to write to the file those lines not selected by the address.

$ cat write_demo3
2,4 !w temp3

$ sed -n -f write_demo3 new

$ cat temp3
Line one.
Five.
This is the sixth sentence.
This is line seven.
Eighth and last.

The following example demonstrates the Next instruction. When it processes the selected line (line 3), sed immediately starts processing the next line without displaying line 3.

$ cat next_demo1
3 n
p

$ sed -n -f next_demo1 new
Line one.
The second line.
This is line four.
Five.
This is the sixth sentence.
This is line seven.
Eighth and last.

The next example uses a textual address. The sixth line contains the string the, so the Next instruction causes sed not to display it.

$ cat next_demo2
/the/ n
p
$ sed -n -f next_demo2 new
Line one.
The second line.
The third.
This is line four.
Five.
This is line seven.
Eighth and last.

The next set of examples uses the file compound.in to demonstrate how sed instructions work together.

$ cat compound. in
1. The words on this page...
2. The words on this page...
3. The words on this page...
4. The words on this page...

The following example substitutes the string words with text on lines 1, 2, and 3 and the string text with TEXT on lines 2, 3, and 4. The example also selects and deletes line 3. The result is text on line 1, TEXT on line 2, no line 3, and words on line 4. The sed utility made two substitutions on lines 2 and 3: text for words and TEXT for text. Then sed deleted line 3.

$ cat compound
1,3 s/words/text/
2,4 s/text/TEXT/
3 d

$ sed -f compound compound.in
1. The text on this page...
2. The TEXT on this page...
4. The words on this page...

The ordering of instructions within a sed program is critical. Both Substitute instructions are applied to the second line in the following example, as in the previous example, but the order in which the substitutions occur changes the result.

$ cat compound2
2,4 s/text/TEXT/
1,3 s/words/text/
3 d

$ sed -f compound2 compound.in
1. The text on this page...
2. The text on this page...
4. The words on this page...

Next compound3 appends two lines to line 2. The sed utility displays all the lines from the file once because no -n option appears on the command line. The Print instruction at the end of the program file displays line 3 an additional time.

$ cat compound3
2 a\
This is line 2a.\
This is line 2b.
3 p

$ sed -f compound3 compound.in
1. The words on this page...
2. The words on this page...
This is line 2a.
This is line 2b.
3. The words on this page...
3. The words on this page...
4. The words on this page...

The next example shows that sed always displays appended text. Here line 2 is deleted but the Append instruction still displays the two lines that were appended to it. Appended lines are displayed even if you use the -n option on the command line.

$ cat compound4
2 a\
This is line 2a.\
This is line 2b.
2 d

$ sed -f compound4 compound.in
1. The words on this page...
This is line 2a.
This is line 2b.
3. The words on this page...
4. The words on this page...

The next example uses regular expressions in the addresses. The regular expression in the following instruction (^.) matches one character at the beginning of every line that is not empty. The replacement string (between the second and third slashes) contains two SPACEs followed by an ampersand (&). The ampersand takes on the value of whatever the regular expression matched.

$ sed 's/^./  &/' new
  Line one.
  The second line.
  The third.
...

This type of substitution is useful for indenting a file to create a left margin. See Appendix A for more information on regular expressions.

You can also use the simpler form s/^/ / to add SPACEs to the beginnings of lines. However, in addition to placing SPACEs at the beginnings of lines with text on them, this instruction places a SPACE at the beginning of every empty linesomething the preceding command does not do.

You may want to put the preceding sed instruction into a shell script so that you do not have to remember it (and retype it) each time you want to indent a file. The chmod utility gives you read and execute permission to the ind file.

$ cat ind
sed 's/^./  &/' $*
$ chmod u+rx ind
$ ind new
  Line one.
  The second line.
  The third.
...

Stand-alone script

When you run the preceding shell script, it creates two processes: It calls a shell, which in turn calls sed. You can eliminate the overhead associated with the shell process by putting the line #!/usr/bin/sed -f (page 267) at the start of the script, which runs the sed utility directly. You need read and execute permission to the file holding the script.

$ cat ind2
#!/usr/bin/sed -f
s/^./\  /

In the following sed program, the regular expression (two SPACEs followed by * $) matches one or more SPACEs at the end of a line. This program removes trailing SPACEs at the ends of lines, which is useful for cleaning up files that you created using vim.

$ cat cleanup
sed 's/ *$//' $*

The cleanup2 script runs the same sed command as cleanup but stands alone: It calls sed directly with no intermediate shell.

$ cat cleanup2
#!/usr/bin/sed -f
s/ *$//

Hold space

The next sed program makes use of the Hold space to exchange pairs of lines in a file.

$ cat s1
# Copy Pattern space (line just read) to Hold space.
h
# Read the next line of input into Pattern space.
n
# Output Pattern space.
p
# Copy Hold space to Pattern space.
g
# Output Pattern space (which now holds the previous line).
p

$ sed -nf s1 new
The second line.
Line one.
This is line four.
The third.
This is the sixth sentence.
Five.
Eighth and last.
This is line seven.

The commands in the s1 program process pairs of input lines. This program reads a line and stores it; reads another line and displays it; and then retrieves the stored line and displays it. After processing a pair of lines the program starts over with the next pair of lines.

The next sed program adds a blank line after each line in the input file (i.e., it double-spaces a file).

$ sed 'G' new
Line one.

The second line.

The third.

This is line four.
...

The G instruction appends a NEWLINE and the contents of the Hold space to the Pattern space. Unless you put something in the Hold space, it is empty. Thus the G instruction copies a NEWLINE to each line of input before sed displays the line(s) from the Pattern space.

The s2 sed program reverses the order of the lines in a file just as the tac utility does.

$ cat s2
# On all but the first line, append a NEWLINE and the
# contents of the Hold space to the Pattern space.
2,$G
# Copy the Pattern space to the Hold space.
h
# Delete all except the last line.
$!d

$ sed -f s2 new
Eighth and last.
This is line seven.
This is the sixth sentence.
Five.
This is line four.
The third.
The second line.
Line one.

This program includes three commands: 2,$G, h, and $!d. To understand this script it is important to understand how the address of the last command works: The $ is the address of the last line of input and the ! negates the address. The result is an address that selects all except the last line of input. In the same fashion you could replace the first command with 1!G: Select all except the first line for processing; the results would be the same.

Here is what happens as s2 processes the new file:

1.
The sed utility reads the first line of input (Line one.) into the Pattern space.

  1. The 2,$G does not process the first line of inputbecause of its address the G instruction starts processing at the second line.

  2. The h copies Line one. from the Pattern space to the Hold space.

  3. The $!d deletes the contents of the Pattern space. Because there is nothing in the Pattern space, sed does not display anything.

2.
The sed utility reads the second line of input (The second line.) into the Pattern space.

  1. The 2,$G adds what is in the Hold space (Line one.) to the Pattern space. The Pattern space now has The second line.NEWLINE Line one.

  2. The h copies what is in the Pattern space to the Hold space.

  3. The $!d deletes the second line of input. Because it is deleted, sed does not display it.

3.
The sed utility reads the third line of input (The third.) into the Pattern space.

  1. The 2,$G adds what is in the Hold space (The second line.NEWLINE Line one.) to the Pattern space. The Pattern space now has The third.NEWLINE The second line.NEWLINE Line one.

  2. The h copies what is in the Pattern space to the Hold space.

  3. The $!d deletes the contents of the Pattern space. Because there is nothing in the Pattern space, sed does not display anything.

...

8.
The sed utility reads the eighth (last) line of input into the Pattern space.

  1. The 2,$G adds what is in the Hold space to the Pattern space. The Pattern space now has all the lines from new in reverse order.

  2. The h copies what is in the Pattern space to the Hold space. This step is not necessary for the last line of input but does not alter the program's output.

  3. The $!d does not process the last line of input. Because of its address the d instruction does not delete the last line.

  4. The sed utility displays the contents of the Pattern space.


Chapter Summary

The sed (stream editor) utility is a batch (noninteractive) editor. It takes its input from files you specify on the command line or from standard input. Unless you redirect the output from sed, it goes to standard output.

A sed program consists of one or more lines with the following syntax:

[address[,address]]instruction [argument-list]

The addresses are optional. If you omit the address, sed processes all lines of input. The instruction is the editing instruction that modifies the text. The addresses select the line(s) the instruction part of the command operates on. The number and kinds of arguments in the argument-list depend on the instruction.

In addition to basic instructions, sed includes some powerful advanced instructions. One set of these instructions allows sed programs to store data temporarily in a buffer called the Hold space. Other instructions provide unconditional and conditional branching in sed programs.


Exercises

1.

Write a sed command that copies a file to standard output, removing all lines that begin with the word Today.

2.

Write a sed command that copies only those lines of a file that begin with the word Today to standard output.

3.

Write a sed command that copies a file to standard output, removing all blank lines (lines with no characters on them).

4.

Write a sed program named ins that copies a file to standard output, changing all occurrences of cat to dog and preceding each modified line with a line that says following line is modified.

5.

Write a sed program named div that copies a file to standard output, copies the first five lines to a file named first, and copies the rest of the file to a file named last.

6.

Write a sed command that copies a file to standard output, replacing a single SPACE as the first character on a line with a 0 (zero) only if the SPACE is immediately followed by a number (09). For example:

abc     abc
 abc     abc
 85c    085c
55b     55b
 000    0000

7.

How can you use sed to triple-space (i.e., add two blank lines after each line in) a file?


Part VI: Command Reference

    Command Reference


Command Reference

The following tables list the utilities covered in this part of the book grouped by function and alphabetically within function. Although most of these are true utilities (programs that are separate from the shells), some are built into the shells (shell builtins). The sample utility on page 664 shows the format of the description of each utility in this part of the book.


Utilities That Display and Manipulate Files

awk | Searches for and processes patterns in a filepage 609
bzip2 | Compresses or decompresses filespage 668
cat | Joins and displays filespage 671
cmp | Compares two filespage 684
comm | Compares sorted filespage 686
cp | Copies filespage 690
cpio | Creates an archive, restores files from an archive, or copies a directorypage 693
cut | Selects characters or fields from input linespage 699
dd | Converts and copies a filepage 703
diff | Displays the differences between two filespage 707
ditto | Copies files and creates and unpacks archivespage 715
find | Finds files based on criteriapage 728
fmt | Formats text very simplypage 736
grep | Searches for a pattern in filespage 751
gzip | Compresses or decompresses filespage 756
head | Displays the beginning of a filepage 759
less | Displays text files, one screen at a timepage 768
ln | Makes a link to a filepage 772
lpr | Sends files to printerspage 774
ls | Displays information about one or more filespage 777
man | Displays documentation for commandspage 788
mkdir | Creates a directorypage 791
mv | Renames or moves a filepage 792
od | Dumps the contents of a filepage 799
open | Opens files, directories, and URLspage 803
paste | Joins corresponding lines from filespage 807
pax | Creates an archive, restores files from an archive, or copies a directorypage 809
plutil | Manipulates property list filespage 815
pr | Paginates files for printingpage 817
rm | Removes a file (deletes a link)page 827
rmdir | Removes a directorypage 829
sed | Edits a file (not interactively)page 641
sort | Sorts and/or merges filespage 837
split | Divides a file into sectionspage 846
strings | Displays strings of printable characterspage 853
tail | Displays the last part of a filepage 859
tar | Stores or retrieves files to/from an archive filepage 862
touch | Changes a file's access and/or modification timepage 877
uniq | Displays unique linespage 885
wc | Displays the number of lines, words, and bytespage 888


Network Utilities

ftp | Transfers files over a networkpage 738
rcp | Copies one or more files to or from a remote systempage 823
rlogin | Logs in on a remote systempage 826
rsh | Executes commands on a remote systempage 830
scp | Securely copies one or more files to or from a remote systempage 832
ssh | Securely executes commands on a remote systempage 847
telnet | Connects to a remote system over a networkpage 868


Utilities That Display and Alter Status

cd | Changes to another working directorypage 673
chgrp | Changes the group associated with a filepage 675
chmod | Changes the access mode (permissions) of a filepage 676
chown | Changes the owner of a file and/or the group the file is associated withpage 682
date | Displays or sets the system time and datepage 701
df | Displays disk space usagepage 705
du | Displays information on disk usage by filepage 718
file | Displays the classification of a filepage 726
finger | Displays information about userspage 734
GetFileInfo | Displays file attributespage 749
kill | Terminates a process by PID numberpage 761
killall | Terminates a process by namepage 763
launchctl | Controls the launchd daemonpage 765
nice | Changes the priority of a commandpage 794
nidump | Displays contents of a NetInfo databasepage 796
nohup | Runs a command that keeps running after you log outpage 798
ps | Displays process statuspage 819
renice | Changes the priority of a processpage 825
SetFile | Sets file attributespage 834
sleep | Creates a process that sleeps for a specified intervalpage 836
stat | Displays information about filespage 851
stty | Displays or sets terminal parameterspage 854
top | Dynamically displays process statuspage 874
umask | Establishes the file-creation permissions maskpage 883
w | Displays information about system userspage 887
which | Shows where in PATH a command is locatedpage 889
who | Displays information about logged-in userspage 890


Utilities That Are Programming Tools

configure | Configures source code automaticallypage 688
gcc | Compiles C, Objective-C, and C++ programspage 745
make | Keeps a set of programs currentpage 783
otool | Displays object, library, and executable filespage 805


Miscellaneous Utilities

at | Executes commands at a specified timepage 665
cal | Displays a calendarpage 670
crontab | Maintains crontab filespage 697
diskutil | Checks, modifies, and repairs local volumespage 712
dmesg | Displays kernel messagespage 717
echo | Displays a messagepage 720
expr | Evaluates an expressionpage 722
sysctl | Displays and alters kernel variablespage 858
tee | Copies standard input to standard output and one or more filespage 867
test | Evaluates an expressionpage 871
tr | Replaces specified characterspage 879
tty | Displays the terminal pathnamepage 882
xargs | Converts standard input into command linespage 892


Standard Multiplicative Suffixes

Several utilities allow you to use the suffixes listed in Table VI-1 following byte counts. You can precede a multiplicative suffix with a number that is a multiplier. For example, 5K means 5 x 210. The absence of a multiplier indicates that the multiplicative suffix is to be multiplied by 1. The utilities that allow these suffixes are marked as such. Utilities that accept only some of these suffixes or accept similar suffixes are identified in the text.

Table VI-1. Multiplicative suffixes

Suffix

Multiplicative value

KB

1,000 (103)

K

1,024 (210)

PB

1015

P

250

MB

1,000,000 (106)

M

1,048,576 (220)

GB

1,000,000,000 (109)

G

1,073,741,824 (230)

TB

1012

T

240

EB

1018

E

260

ZB

1021

Z

270

YB

1024

Y

280

BLOCKSIZE

Some utilities use the BLOCKSIZE environment variable to set a default block size. You can set BLOCKSIZE to a value that is a number of bytes or to a value that uses one of the K, M, or G suffixes. The text identifies utilities that use BLOCKSIZE.


Common Options

Most GNU utilities share the options listed in Table VI-2. The utilities that use these options are marked as such. Almost all POSIX utilities support the and options.

Table VI-2. Common command line options

Option

Effect

A single hyphen appearing in place of a filename indicates that the utility will accept standard input in place of the file.

A double hyphen marks the end of the options on a command line. You can follow this option with an argument that begins with a hyphen. Without this option the utility assumes that an argument that begins with a hyphen is an option.

help

Displays a help message for the utility.

version

Displays version information for the utility.


The sample Utility

The following description of the sample utility shows the format that is used to describe the utilities in this part of the book. These descriptions are similar to the man page descriptions (pages 28 and 788); however, most users find the descriptions in this book easier to read and understand. These descriptions emphasize the most useful features of the utilities and often leave out the more obscure features. For information about the less commonly used features, refer to the man and info pages or call the utility with the help option, which works with some utilities.

sample: Very brief description of what the utility does

sample [options] arguments

Following the syntax line is a description of the utility. The syntax line shows how to run the utility from the command line. Options and arguments enclosed in brackets ([]) are not required. Words that appear in this italic typeface must be typed as is. Words that you must substitute when you type appear in this bold italic typeface. Words listed as arguments to a command identify single arguments (for example, source-file) or groups of similar arguments (for example, directory-list).

Arguments

This section describes the arguments that you can use when you run the utility. The argument itself, as shown in the preceding syntax line, is printed in this bold italic typeface.

Options

This section lists some of the options you can use with the command. Unless otherwise specified, you must precede options with one or two hyphens. Most commands accept a single hyphen before multiple options (page 115). Options in this section are ordered alphabetically by short (single-hyphen) options. If an option has only a long version (two hyphens), it is ordered by its long option. Following are some sample options:

make-dirs | d | This option has a long and a short version. You can use either option; they are equivalent.
delimiter=dchar |  |
 | d | dchar
 |  | This option includes an argument. The argument is set in a bold italic typeface in both the heading and the description. You substitute another word (filename, string of characters, or other value) for any arguments you see in this typeface. Type characters that are in bold type (such as the delimiter and d) as is, letter for letter.
 | t | (table of contents) This example shows a simple option preceded by a single hyphen and not followed with any arguments. It has no long version. The table of contents appearing in parentheses at the beginning of the description is a cue, suggestive of what the option letter stands for.

Discussion

This optional section contains a discussion about how to use the utility and any quirks it may have.

Notes

This section contains miscellaneous notessome important and others merely interesting.

Examples

This section contains examples of how to use the utility. This section is tutorial and is more casual than the preceding sections of the description.

at: Executes commands at a specified time

at [options] time [date | +increment]

atq

atrm job-list

batch [options] [time]

The at and batch utilities execute commands at a specified time. They accept commands from standard input or, with the f option, from a file. Commands are executed in the same environment as the at or batch command. Unless redirected, standard output and standard error from commands are emailed to the user who ran at or batch. A job is the group of commands that is executed by one call to at. The batch utility differs from at in that it schedules jobs so that they run when the CPU load on the system is low.

The atq utility displays a list of at jobs you have queued; atrm cancels pending at jobs.

Arguments

The time is the time of day when at runs the job. You can specify the time as a one-, two-, or four-digit number. One- and two-digit numbers specify an hour, and four-digit numbers specify an hour and minute. You can also give the time in the form hh:mm. The at utility assumes a 24-hour clock unless you place am or pm immediately after the number, in which case it uses a 12-hour clock. You can also specify time as now, midnight, noon, or teatime (4:00 PM).

The date is the day of the week or day of the month on which at is to execute the job. When you do not specify a day, at executes the job today if the hour you specify in time is greater than the current hour. If the hour is less than the current hour, at executes the job tomorrow.

You specify a day of the week by spelling it out or abbreviating it to three letters. You can also use the words today and tomorrow. Use the name of a month followed by the number of the day in the month to specify a date. You can follow the month and day number with a year.

The increment is a number followed by one of the following (plural and singular are both allowed): minutes, hours, days, or weeks. The at utility adds the increment to time. You cannot specify an increment for a date.

When using atrm, job-list is a list of one or more at job numbers. You use atq to list job numbers.

Options

f | file
 | Specifies that commands come from file instead of standard input. This option is useful for long lists of commands or commands that are executed repeatedly.
m | (mail) Sends you email after a job is run, even when nothing is sent to standard output or standard error. When a job generates output, at always emails it to you, regardless of this option.

Notes

The shell saves the environment variables and the working directory at the time you submit an at job so that they are available when at executes commands.

Jobs you submit using at are run by the at daemon (atrun). This daemon stores jobs in /var/at/jobs and output in /var/at/spool.

Under OS X 10.4 and later, the atrun daemon is disabled by default for power conservation. Working as a privileged user, you can enable and disable atrun with the following commands:

# launchctl load -w /System/Library/LaunchDaemons/com.apple.atrun.plist

# launchctl unload -w /System/Library/LaunchDaemons/com.apple.atrun.plist

See launchctl (page 765) for more information.

Examples

You can use any of the following techniques to paginate and print long_file tomorrow at 2:00 AM. The first example executes the command directly from the command line; the last two examples use the pr_tonight file, which contains the necessary command, and execute it using at.

$ at 2am
pr long_file  |  lpr
CONTROL-D
job 8 at Wed  Jun 22 02:00:00 2005

$ cat pr_tonight
#!/bin/bash
pr long_file  |  lpr

$ at -f pr_tonight 2am
job 9 at Wed Jun 22 02:00:00 2005

$ at 2am < pr_tonight
job 10 at Wed Jun 22 02:00:00 2005

If you execute commands directly from the command line, you must signal the end of the commands by pressing CONTROL-D at the beginning of a line. After you press CONTROL-D, at displays a line that begins with job followed by the job number and the time when at will execute the job.

If you run atq after the preceding commands, it displays a list of jobs in its queue:

$ atq
8       Wed Jun 22 02:00:00 2005
9       Wed Jun 22 02:00:00 2005
10      Wed Jun 22 02:00:00 2005

The following command removes job number 9 from the queue:

$ atrm 9
$ atq
8       Wed Jun 22 02:00:00 2005
10      Wed Jun 22 02:00:00 2005

The next example executes cmdfile at 3:30 PM (1530 hours) one week from today:

$ at -f cmdfile 1530 +1 week
job 12 at Wed Jun 29 15:30:00 2005

Next at executes a job at 7:00 PM on Thursday. This job uses find to create an intermediate file, redirects the output sent to standard error, and prints the file.

$ at 7pm Thursday
find / -name "*.c" >report.out 2>report.err
lpr report.out
CONTROL-D
job 13 at Thu Jun 23 19:00:00 2005

bzip2: Compresses or decompresses files

bzip2 [options] [file-list]

bunzip2 [options] [file-list]

bzcat [options] [file-list]

bzip2recover [file]

The bzip2 utility compresses files; bunzip2 restores files compressed with bzip2; bzcat displays files compressed with bzip2.

Arguments

The file-list specifies the pathnames of one or more files (no directories) that are to be compressed or decompressed. If file-list is empty or if the special option is present, bzip2 reads from standard input. The stdout option causes bzip2 to write to standard output.

Options

Accepts the common options described on page 663.

stdout | c | Writes the results of compression or decompression to standard output.
decompress | d | Decompresses a file compressed with bzip2. The combination of this option with bzip2 is equivalent to the bunzip2 command.
fast or best | n | Sets the block size when compressing a file. The n is a digit from 1 to 9, where 1 (fast) generates a block size of 100 kilobytes and 9 (best) generates a block size of 900 kilobytes. The default level is 9. The options fast and best are provided for compatibility with gzip and do not necessarily yield the fastest or best compression.
force | f | Forces compression even if a file already exists, has multiple links, or comes directly from a terminal. The option has a similar effect with bunzip2.
keep | k | Does not delete input files while compressing or decompressing them.
quiet | q | Suppresses warning messages; does display critical messages.
test | t | Verifies the integrity of a compressed file. Displays nothing if the file is OK.
verbose | v | For each file being compressed displays the name of the file, the compression ratio, the percentage of space saved, and the sizes of the decompressed and compressed files.

Discussion

The bzip2 and bunzip2 utilities work similarly to gzip and gunzip; see the discussion of gzip (page 757) for more information. Normally bzip2 does not overwrite a file; you must use force to overwrite a file during compression or decompression.

Notes

The bzip2 home page is sources.redhat.com/bzip2.

The bzip2 utility does a better job of compressing files than gzip does.

Use the bzip modifier with tar (page 864) to compress archive files with bzip2.

bzcat file-list

Works like cat except that it uses bunzip2 to decompress file-list as it copies files to standard output.

bzip2recover

Attempts to recover a damaged file that was compressed with bzip2.

Examples

In the following example, bzip2 compresses a file and gives the resulting file the same name with a .bz2 filename extension. The v option displays statistics about the compression.

$ ls -l
total 728
-rw-r--r--  1 sam sam 737414 Feb 20 19:05 bigfile
$ bzip2 -v bigfile
  bigfile: 3.926:1, 2.037 bits/byte, 74.53% saved, 737414 in, 187806 out
$ ls -l
total 188
-rw-r--r--  1 sam sam 187806 Feb 20 19:05 bigfile.bz2

Next touch creates a file with the same name as the original file; bunzip2 refuses to overwrite the file in the process of decompressing bigfile.bz2. The force option enables bunzip2 to overwrite the file.

$ touch bigfile
$ bunzip2 bigfile.bz2
bunzip2: Output file bigfile already exists.
$ bunzip2 --force bigfile.bz2
$ ls -l
total 728
-rw-r--r--  1 sam sam 737414 Feb 20 19:05 bigfile

cal: Displays a calendar

cal [options] [[month] year]

The cal utility displays a calendar for a month or a year.

Arguments

The arguments specify the month and year for which cal displays a calendar. The month is a decimal integer from 1 to 12 and the year is a decimal integer. Without any arguments, cal displays a calendar for the current month. When you specify a single argument, it is assumed to be the year.

Options

j | (Julian) Displays a Julian calendara calendar that numbers the days consecutively from January 1 (1) through December 31 (365 or 366).
y | (year) Displays a calendar for the current year.

Notes

Do not abbreviate the year. The year 05 is not the same as 2005.

The cal utility switches to the Gregorian calendar in September 1752.

Examples

The following command displays a calendar for August 2007:

$ cal 8 2007
     August 2007
Su Mo Tu We Th Fr Sa
          1  2  3  4
 5  6  7  8  9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31

Next is a Julian calendar for 1949:

$ cal -j 1949
                            1949

           January                       February
  S   M   Tu   W   Th   F   S    S   M   Tu   W   Th   F   S
                            1            32   33  34  35  36
  2   3    4   5    6   7   8   37  38   39  40   41  42  43
  9  10   11  12   13  14  15   44  45   46  47   48  49  50
 16  17   18  19   20  21  22   51  52   53  54   55  56  57
 23  24   25  26   27  28  29   58  59
 30  31
...

cat: Joins and displays files

cat [options] [file-list]

The cat utility copies files to standard output. You can use cat to display the contents of one or more text files on the screen.

Arguments

The file-list specifies the pathnames of one or more files that cat processes. If you do not specify an argument or if you specify a hyphen (-) in place of a filename, cat reads from standard input.

Options

b | (blank) Numbers all lines that are not blank as they are written to standard output.
e | (end) Marks the ends of lines with dollar signs. Implies v.
n | (number) Numbers all lines as they are written to standard output.
s | (squeeze blanks) Removes extra blank lines so there are never two or more blank lines in a row.
t | (tab) Displays TABs as ^I. Implies v.
v | (verbose) Displays CONTROL characters with the caret notation (^M) and displays characters that have the high bit set (META characters) with the M- notation. This option does not convert TABs and LINEFEEDs. Use t if you want to display TABs as ^I. LINEFEEDs cannot be displayed as anything but themselves; otherwise, the line would be too long.

Notes

See page 121 for a discussion of cat, standard input, and standard output.

Use the od utility (page 799) to display the contents of a file that does not contain text (for example, an executable program file).

The name cat is derived from one of the functions of this utility, catenate, which means to join together sequentially, or end to end.

Caution: Set noclobber to avoid overwriting a file

Despite cat's warning message, the shell destroys the input file (letter) before invoking cat in the following example:

$ cat memo letter > letter
cat: letter: input file is output file

You can prevent the file from being overwritten in this situation by setting the noclobber variable (pages 125 and 363).

Examples

The following command displays the contents of the memo text file on the terminal:

$ cat memo
...

The next example catenates three text files and redirects the output to the all file:

$ cat page1 letter memo > all

You can use cat to create short text files without using an editor. Enter the following command line, type (or paste) the text you want in the file, and press CONTROL-D on a line by itself:

$ cat > new_file
...
(text)
 ...
CONTROL-D

In this case cat takes input from standard input (the keyboard) and the shell redirects standard output (a copy of the input) to the file you specify. The CONTROL-D signals EOF (end of file) and causes cat to return control to the shell (page 122).

In the next example, a pipe sends the output from who to standard input of cat. The shell redirects cat's output to the file named output. After the commands have finished executing, output contains the contents of the header file, the output of who, and footer. The hyphen on the command line causes cat to read standard input after reading header and before reading footer.

$ who | cat header - footer > output

cd: Changes to another working directory

cd [options] [directory]

The cd builtin makes directory the working directory.

Arguments

The directory is the pathname of the directory that is to be the new working directory. Without an argument, cd makes your home directory the working directory. Using a hyphen in place of directory changes to the previous working directory.

Notes

The cd command is a bash and tcsh builtin.

See page 82 for a discussion of cd.

Without an argument, cd makes your home directory the working directory; it uses the value of the HOME (bash, page 284) or home (tcsh, page 359) variable for this purpose.

With an argument of a hyphen, cd makes the previous working directory the working directory. It uses the value of the OLDPWD (bash) or owd (tcsh) variable for this purpose.

The CDPATH (bash, page 290) or cdpath (tcsh, page 359) variable contains a colon-separated list of directories that cd searches. Within the list a null directory name (::) or a period (:.:) represents the working directory. If CDPATH or cdpath is not set, cd searches only the working directory for directory. If this variable is set and directory is not an absolute pathname (does not begin with a slash), cd searches the directories in the list; if the search fails, cd searches the working directory. See page 290 for a discussion of CDPATH.

Examples

The following cd command makes Alex's home directory his working directory. The pwd builtin verifies the change:

$ pwd
/Users/alex/literature
$ cd
$ pwd
/Users/alex

The next command makes the /Users/alex/literature directory the working directory:

$ cd /Users/alex/literature
$ pwd
/Users/alex/literature

Next the cd utility makes a subdirectory of the working directory the new working directory:

$ cd memos
$ pwd
/Users/alex/literature/memos

Finally cd uses the .. reference to the parent of the working directory to make the parent the new working directory:

$ cd ..
$ pwd
/Users/alex/literature

chgrp: Changes the group associated with a file

chgrp [options] group file-list

The chgrp utility changes the group associated with one or more files.

Arguments

The group is the name or numeric group ID of the new group. The file-list specifies the pathnames of one or more files whose group association is to be changed.

Options

f | (force) Prevents the display of warning messages about files whose permissions prevent you from changing their group IDs.
H | (dereference) Follows the files that symbolic links given on the command line point to, not the symbolic links themselves. Does not follow the files that symbolic links found while descending a directory hierarchy point to. Works only with R.
L | (dereference) Follows the files that symbolic links point to, not the symbolic links themselves. Includes the files that symbolic links found while descending a directory hierarchy point to. Works only with R.
P | (no dereference) Does not follow symbolic links while descending a directory hierarchy (default). Works only with R.
R | (recursive) Recursively descends a directory specified in file-list and changes the group ID on all files in the directory hierarchy.
v | (verbose) Displays for each file a message saying whether its group was retained or changed.

Notes

Only the owner of a file or root can change the group association of a file. Also, unless you are root, you must belong to the specified group to change the group ID of a file to that group.

See page 682 for information on how chown can change the group associated with, as well as the owner of, a file.

Examples

The following command changes the group that the manuals file is associated with; the new group is pubs.

$ chgrp pubs manuals

chmod: Changes the access mode (permissions) of a file

chmod [options] who operator permission file-list (symbolic)

chmod [options] mode file-list (absolute)

The chmod utility changes the ways in which a file can be accessed by the owner of the file, the group the file is associated with, and/or all other users. Only the owner of a file or Superuser can change the access mode, or permissions, of a file. You can specify the new access mode absolutely or symbolically.

Arguments

The who, operator, and permission arguments specify symbolically which modes will be changed in what ways. The mode argument specifies the same information absolutely. The file-list specifies the pathnames of one or more files whose permissions chmod is to change.

Symbolic

You can specify multiple sets of symbolic modes (who operator permission) by separating each set from the next with a comma.

The chmod utility changes the access permission for the class of users specified by who. The class of users is designated by one or more of the letters specified in the who column of Table VI-3

Table VI-3. Symbolic mode user class specification

who

User class

Meaning

u

User

Owner of the file

g

Group

Group to which the owner belongs

o

Other

All other users

a

All

Can be used in place of ugo

Table VI-4 lists the symbolic mode operators.

Table VI-4. Symbolic mode operators

operator

Meaning

+

Adds permission for the specified user class

-

Removes permission for the specified user class

=

Sets permission for the specified user classresets all other permissions for that user class

The access permission is specified by one or more of the letters listed in Table VI-5.

Table VI-5. Symbolic mode permissions

permission

Meaning

r

Sets read permission

w

Sets write permission

x

Sets execute permission

s

Sets user ID or group ID (depending on the who argument) to that of the owner of the file while the file is being executed (For more information see page 90.)

t

Sets the sticky bit (Only Superuser can set the sticky bit, and it can be used only with u; see page 955.)

X

Makes the file executable only if it is a directory or if another user class has execute permission

u

Sets the specified permissions to those of the owner

g

Sets the specified permissions to those of the group

o

Sets the specified permissions to those of others

Absolute

You can use an octal number to specify the access mode. Construct the number by ORing the appropriate values from Table VI-6. To OR two octal numbers from this table, just add them. (Refer to Table VI-7 for examples.)

Table VI-6. Absolute mode specifications

mode

Meaning

4000

Sets user ID when the program is executed (page 90)

2000

Sets group ID when the program is executed (page 90)

1000

Sticky bit (page 955)

0400

Owner can read the file

0200

Owner can write to the file

0100

Owner can execute the file

0040

Group can read the file

0020

Group can write to the file

0010

Group can execute the file

0004

Others can read the file

0002

Others can write to the file

0001

Others can execute the file

Table VI-7. Examples of absolute mode specifications

Mode

Meaning

0777

Owner, group, and others can read, write, and execute file

0755

Owner can read, write, and execute file; group and others can read and execute file

0711

Owner can read, write, and execute file; group and others can execute file

0644

Owner can read and write file; group and others can read file

0640

Owner can read and write file; group can read file; and others cannot access file

Table VI-7 lists some typical modes.

ACLs

Although ACLs (Access Control Lists, page 97) are most often modified using the OS X GUI, you can also use chmod to work with them. A chmod command to create, modify, or delete an access rule adheres to the following format:

chmod option user|group allow|deny action

where option is +a (add rule), a (remove rule), or =a (change rule); user is a username or UID number; group is a group name or group ID number; and action is one or more of read, write, append, and execute. For example, the following command adds a rule that gives Jenny read and write access to the memo file:

$ chmod +a "jenny allow read,write" memo

If you replace allow with deny in the preceding command, the rule prohibits Jenny from reading from or writing to memo.

The kernel processes multiple rules in an ACL in the order displayed by ls le. The order entries are stored in is not necessarily the order they were added in. In general, deny entries come before allow entries. You can override the default order by adding a rule with +a# n. This rule will be added as rule number n.

There are two ways to remove access rules. First, you can specify a rule using the same format that you used when you added it. The following command removes the rule that was added in the previous example:

$ chmod a "jenny allow read,write" memo

Second, after using ls le to display a file's ACL, you can use chmod a# n to remove rule number n.

Options

f | (force) Prevents the display of warning messages about files whose permissions prevent chmod from changing the permissions of the file.
H | (dereference) Follows the files that symbolic links given on the command line point to, not the symbolic links themselves. Does not follow the files that symbolic links found while descending a directory hierarchy point to. Works only with R.
L | (dereference) Follows the files that symbolic links point to, not the symbolic links themselves. Includes the files that symbolic links found while descending a directory hierarchy point to. Works only with R.
P | (no dereference) Does not follow symbolic links while descending a directory hierarchy (default). Works only with R.
R | (recursive) Recursively descends a directory specified in file-list and changes the permissions on all files in the directory hierarchy.
v | (verbose) Displays the name of each file as its permissions are changed.

Notes

When you use symbolic arguments, you can omit the permission from the command line only when the operator is =. This omission takes away all permissions. See the second example in the next section.

See page 87 for a discussion of access permissions in general and page 90 for information about how directory permissions work.

Examples

The following examples show how to use chmod to change the permissions of the file named temp. The initial access mode of temp is shown by ls (see page 87 for information about the ls display):

$ ls -l temp
-rw-rw-r-- 1 alex   pubs   57 Jul 12 16:47 temp

When you do not follow an equal sign with a permission, chmod removes all permissions for the specified user class. The following command removes all access permissions for the group and all other users so that only the owner has access to the file:

$ chmod go= temp
$ ls -l temp
-rw-------   1 alex   pubs   57 Jul 12 16:47 temp

The next command changes the access modes for all users (owner, group, and others) to read and write. Now anyone can read from or write to the file.

$ chmod a=rw temp
$ ls -l temp
-rw-rw-rw-   1 alex  pubs   57 Jul 12 16:47 temp

Using an absolute argument, a=rw becomes 666. The next command performs the same function as the previous one:

$ chmod 666 temp

The next command removes write access permission for other users. As a result Alex and members of the pubs group can still read from and write to the file, but other users can only read from the file:

$ chmod o-w temp
$ ls -l temp
-rw-rw-r--   1 alex  pubs  57 Jul 12 16:47 temp

The following command yields the same result, using an absolute argument:

$ chmod 664 temp

The next command adds execute access permission for all users:

$ chmod a+x temp
$ ls -l temp
-rwxrwxr-x   1 alex  pubs  57 Jul 12 16:47 temp

If temp is a shell script or other executable file, all users can now execute it. (You need read and execute access to execute a shell script but only execute access to execute a binary file.) The following absolute command yields the same result:

$ chmod 775 temp

The next command uses symbolic arguments to achieve the same result as the preceding one. It sets permissions to read, write, and execute for the owner and group, and to read and write for other users. A comma separates the sets of symbolic modes.

$ chmod ug=rwx,o=rw temp

ACLs

The next series of examples manipulate ACLs (page 97), using ls le to display the results. First Alex gives Jenny permission to read from and write to the memo file. The plus sign (+) to the right of the permissions in the output of ls indicates that the file has an ACL.

$ ls -le memo
-rw-r--r-- + 1 alex  alex  19 Jun 24 15:30 memo
$ chmod +a "jenny allow read,write" memo
$ ls -le memo
-rw-r--r-- + 1 alex  alex  19 Jun 24 15:30 memo
 0: user:jenny allow read,write

Alex then decides to remove the write permissions he just gave Jenny. If the access specification given to a a option is only part of an existing rule, the remainder of the rule remains.

$ chmod -a "jenny allow write" memo
$ ls -le memo
-rw-r--r-- + 1 alex  alex  19 Jun 24 15:30 memo
 0: user:jenny allow read

The next example removes the ACL entry entirely, specifying it by number. Although there are no rules, the file is marked with a plus sign to indicate that it still has an ACL.

$ chmod -a# 0 memo
$ ls -le memo
-rw-r--r-- + 1 alex  alex  19 Jun 24 15:30 memo

When you specify an ACL operation that it cannot complete, chmod returns an error. The following multiline command uses a while loop (page 540) to remove all ACL rules for memo. The : (null) builtin provides the loop body.

$ while chmod -a# 0 memo
> do :
> done
chmod: No ACL present: Invalid argument

The next example shows that chmod inserts rules in an ACL in a default order:

$ chmod +a "jenny allow read" memo
$ chmod +a "jenny deny read" memo
$ ls -le memo
-rw-r--r-- + 1 alex  alex  19 Jul 19 21:59 memo
 0: user:jenny deny read
 1: user:jenny allow read

Even though the allow rule was added first, the deny rule appears before it. Next Alex gives read permission to Sam, Max, and Zach:

$ chmod +a "sam allow read" memo
$ chmod +a "max allow read" memo
$ chmod +a "zach allow read" memo
$ ls -le memo
-rw-r--r-- + 1 alex  alex  19 Jul 19 21:59 memo
 0: user:jenny deny read
 1: user:jenny allow read
 2: user:sam allow read
 3: user:max allow read
 4: user:zach allow read

After giving these permissions, Alex realizes that he meant to give Sam write access, too. He could create a new rule by using sam allow write, but he would like to replace Sam's rule with the correct one. The following command replaces (=a) rule number 2 (# 2):

$ chmod =a# 2 "sam allow read,write"
$ ls -le
-rw-r--r-- + 1 alex  alex  19 Jul 19 21:59 memo
 0: user:jenny deny read
 1: user:jenny allow read
 2: user:sam allow read,write
 3: user:max allow read
 4: user:zach allow read

chown: Changes the owner of a file and/or the group the file is associated with

chown [options] owner file-list

chown [options] owner:group file-list

chown [options] owner: file-list

chown [options] :group file-list

The chown utility changes the owner of a file and/or the group the file is associated with. Only root can change the owner of a file. Only root or the owner of a file who belongs to the new group can change the group a file is associated with.

Arguments

The owner is the username or numeric user ID of the new owner. The file-list specifies the pathnames of one or more files whose ownership and/or group association chown is to change. The group is the group name or numeric group ID of the new group that the file is associated with. Table VI-8 shows the ways you can specify the new owner and/or group.

Table VI-8. Specifying the new owner and/or group

Argument

Meaning

owner

The new owner of file-list; the group is not changed

owner:group

The new owner and new group association of file-list

owner:

The new owner of file-list; the group association is changed to that of the new owner's login group

:group

The new group associated with file-list; the owner is not changed

Options

f | (force) Prevents chown from displaying error messages when it is unable to change the ownership/group of a file.
H | (dereference) Follows the files that symbolic links given on the command line point to, not the symbolic links themselves. Does not follow the files that symbolic links found while descending a directory hierarchy point to. Works only with R.
L | (dereference) Follows the files that symbolic links point to, not the symbolic links themselves. Includes the files that symbolic links found while descending a directory hierarchy point to. Works only with R.
P | (no dereference) Does not follow symbolic links while descending a directory hierarchy (default). Works only with R.
R | (recursive) When you include directories in the file-list, this option descends the directory hierarchy, setting the specified ownership/group for all files encountered.
v | (verbose) Displays the name of each file as its ownership is changed.

Examples

The following command changes the owner of the chapter1 file in the manuals directory. The new owner is Jenny.

# chown jenny manuals/chapter1

The following command makes Alex the owner of, and Alex's login group the group associated with, all files in the /Users/alex/literature directory and in all its subdirectories:

# chown -R alex: /Users/alex/literature

The next command changes the ownership of the files in literature to alex and the group associated with these files to pubs:

# chown alex:pubs /Users/alex/literature/*

The final example changes the group association of the files in manuals to pubs without altering their ownership. The owner of the files, who is executing this command, must belong to the pubs group.

$ chown :pubs manuals/*

cmp: Compares two files

cmp [options] file1 [file2 [skip1 [skip2]]]

The cmp utility displays the differences between two files on a byte-by-byte basis. If the files are the same, cmp is silent. If the files differ, cmp displays the byte and line number of the first difference.

Arguments

The file1 and file2 are pathnames of the files that cmp compares. If file2 is omitted, cmp uses standard input instead. Using a hyphen () in place of file1 or file2 causes cmp to read standard input instead of that file.

The skip1 and skip2 are decimal numbers indicating the number of bytes to skip in each file before beginning the comparison.

Options

Accepts the common options described on page 663.

printbytes | b | Displays more information, including filenames, byte and line number, and octal and ASCII values of the first differing byte.
ignoreinitial=n | i | n
 |  | Skips the first n bytes in both files before beginning the comparison.
verbose | l | (lowercase "l") Instead of stopping at the first byte that differs, continues comparing the two files and displays both the location and the value of each byte that differs. Locations are displayed as decimal byte count offsets from the beginning of the files; byte values are displayed in octal. The comparison terminates when an EOF is encountered on either file.
silent or quiet | s | Suppresses output from cmp; only sets the exit status (see "Notes").

Notes

Unlike diff (page 707), cmp works with binary as well as ASCII files. Byte and line numbering start at 1.

The cmp utility does not display a message if the files are identical; it just sets the exit status. This utility returns an exit status of 0 if the files are the same and 1 if they differ. An exit status greater than 1 means that an error occurred.

When you use skip1 (and skip2), the offset values cmp displays are based on the byte where the comparison began. You can use the standard multiplicative suffixes after skip1 and skip2; see Table VI-1 on page 662.

The cmp utility compares data forks (page 93) of a file only. If you want to compare resource forks, you can manually compare the ..namedfork/rsrc files (page 94) for the target files.

Examples

The examples use the files a and b shown below. These files have two differences. The first difference is that the word lazy in file a is replaced by lasy in file b. The second difference is more subtle: A TAB character appears just before the NEWLINE character in file b.

$ cat a
The quick brown fox jumped over the lazy dog.
$ cat b
The quick brown fox jumped over the lasy dog.TAB

The first example uses cmp without any options to compare the two files. The cmp utility reports that the files are different and identifies the offset from the start of the files where the first difference is found:

$ cmp a b
a b differ: char 39, line 1

You can display the values of the bytes at that location by adding the printbytes option:

$ cmp --print-bytes a b
a b differ: char 39, line 1 is 172 z 163 s

The l option displays all bytes that differ between the two files. Because this option creates a lot of output if the files have many differences, you may want to redirect the output to a file or pager. The following example shows the two differences between files a and b. The b option displays the values for the bytes as well. Where file a has a CONTROLJ (NEWLINE), file b has a CONTROLI (TAB). The message saying that it has reached the end of file on file a indicates that file b is longer than file a.

$ cmp -lb a b
39 172 z    163 s
46  12 ^J    11 ^I
cmp: EOF on a

In the next example, the ignoreinitial option is used to skip over the first difference in the files. The cmp utility now reports the second difference. The difference is put at character 7, which is the 46th character in the original file b (7 characters past the ignored 39 characters).

$ cmp --ignore-initial=39 a b
a b differ: char 7, line 1

You can use skip1 and skip2 in place of the ignoreinitial option used in the preceding example:

$ cmp a b 39 39
a b differ: char 7, line 1

comm: Compares sorted files

comm [options] file1 file2

The comm utility displays a line-by-line comparison of two sorted files. The first of the three columns it displays lists the lines found only in file1, the second column lists the lines found only in file2, and the third lists the lines common to both files.

Arguments

The file1 and file2 are pathnames of the files that comm compares. Using a hyphen (-) in place of file1 or file2 causes comm to read standard input instead of that file.

Options

You can combine the options.

1 | Does not display column 1 (does not display lines found only in file1).
2 | Does not display column 2 (does not display lines found only in file2).
3 | Does not display column 3 (does not display lines found in both files).

Notes

If the files have not been sorted, comm will not work properly.

Lines in the second column are preceded by one TAB, and those in the third column are preceded by two TABs.

The exit status indicates whether comm completed normally (0) or abnormally (not 0).

Examples

The following examples use two files, c and d, in the working directory. As with all input to comm, the files have already been sorted:

$ cat c
bbbbb
ccccc
ddddd
eeeee
fffff
$ cat d
aaaaa
ddddd
eeeee
ggggg
hhhhh

Refer to sort on page 837 for information on sorting files.

The following example calls comm without any options, so it displays three columns. The first column lists those lines found only in file c, the second column lists those found only in d, and the third lists the lines found in both c and d:

$ comm c d
        aaaaa
bbbbb
ccccc
                ddddd
                eeeee
fffff
        ggggg
        hhhhh

The next example shows the use of options to prevent comm from displaying columns 1 and 2. The result is column 3, a list of the lines common to files c and d:

$ comm -12 c d
ddddd
eeeee

configure: Configures source code automatically

./configure options

The configure script is part of the GNU Configure and Build System. Software developers who supply source code for their products face the problem of making it easy for relatively naive users to build and install their software package on a wide variety of machine architectures, operating systems, and system software. To solve this problem many software developers supply a shell script named configure with their source code.

When you run configure, it determines the capabilities of the local system. The data collected by configure is used to build the makefiles with which make (page 783) builds the executables and libraries. You can adjust the behavior of configure by specifying command line options and environment variables.

Options

disable-feature | Works in the same manner as enable-feature except that it disables support for feature.
enable-feature | Replace feature with the name of a feature that can be supported by the software being configured. For example, configuring the Z Shell source code with the command configure enable-zsh-mem configures the source code to use the special memory allocation routines provided with zsh instead of using the system memory allocation routines. Check the README file supplied with the software distribution to see the choices available for feature.
help | Displays a detailed list of all options available for use with configure. The contents of this list depend on the software distribution being configured.
prefix=directory | By default configure builds makefiles that install software in the /usr/local directory (when you give the command make install). To install the software into a different directory, replace directory with the pathname of the new target directory.
with-package | Replace package with the name of an optional package that can be included with the software you are configuring. For example, if you configure the source code for the Windows emulator wine with the command configure with-dll, the source code is configured to build a shared library of Windows emulation support. Check the README file supplied with the software distribution to see the choices available for package. Also, configure help usually displays the choices available for package.

Discussion

The GNU Configure and Build System allows software developers to distribute software that can configure itself to be built on a variety of systems. It builds a shell script named configure, which prepares the software distribution to be built and installed on a local system. The configure script searches the local system to find the various dependencies for the software distribution and constructs the appropriate makefiles. Once you have run configure, you can build the software with a make command and install the software with a make install command.

The configure script determines which C compiler to use (usually gcc) and specifies a set of flags to pass to that compiler. You can set the environment variables CC and CFLAGS to override these values with your own choices. (See "Macros" on page 494 and the "Examples" section following for more information.)

Notes

Each package that uses the GNU autoconfiguration utility provides its own custom copy of configure, which the software developer created using the GNU autoconf utility. Read the README and INSTALL files that are provided with the package to obtain detailed information about the available options.

The configure scripts are self-contained and run correctly on a wide variety of systems, including OS X. You do not need any special system resources to use configure.

Examples

The simplest way to call configure is to cd to the base directory for the software distribution you want to configure and then run the following command:

$ ./configure

The ./ is prepended to the command name to ensure that you are running the configure script that was supplied with the software distribution. To cause configure to build makefiles that pass the flags Wall and O2 to gcc, use the following command from bash:

$ CFLAGS="-Wall -02" ./configure

If you are using tcsh, use the following command:

tcsh $ env CFLAGS="-Wall -02" ./configure

cp: Copies files

cp [options] source-file destination-file

cp [options] source-file-list destination-directory

The cp utility copies one or more files. It can make a copy of a single file (first format), or it can copy one or more files to a directory (second format). With the r option, cp can copy directories.

Arguments

The source-file is the pathname of the file that cp makes a copy of. The destination-file is the pathname that cp assigns to the resulting copy of the file.

The source-file-list specifies one or more pathnames of files that cp makes copies of. The destination-directory is the pathname of the directory in which cp places the copied files. With this format, cp gives each of the copied files the same simple filename as its source-file.

The r option enables cp to copy directories recursively from the source-file-list into the destination-directory.

Options

f | (force) When the destination-file exists and cannot be opened for writing, this option causes cp to try to remove destination-file before copying source-file. This option is useful when the user copying a file does not have write permission to an existing destination-file but has write permission to the directory containing the destination-file.
i | (interactive) Prompts you whenever cp would overwrite a file. If you respond with a string that starts with y or Y, cp continues. If you enter anything else, cp does not copy the file.
H | (dereference) Copies the files that symbolic links given on the command line point to, not the symbolic links themselves. Does not copy the files that symbolic links found while descending a directory hierarchy point to. See P.
L | (dereference) Copies the files that symbolic links point to, not the symbolic links themselves. Includes the files that symbolic links found while descending a directory hierarchy point to. See P.
p | (preserve) Creates a destination-file with the same owner, group, permissions, access date, and modification date as the source-file.
P | (no dereference) Copies symbolic links, not the files that the links point to. Without the R option, the default behavior is to dereference links (copy the files that links point to, not the links). With one of these options, cp does not dereference symbolic links (it copies the links, not the files that the links point to).
R | Recursively copies directory hierarchies including ordinary files. The P option is implied.
v | (verbose) Displays the name of each file as cp copies it.

Notes

If the destination-file exists before you execute cp, cp overwrites the file, destroying the contents but leaving the access privileges, owner, and group associated with the file as they were.

If the destination-file does not exist, cp uses the access privileges of the source-file. The user who copies the file becomes the owner of the destination-file and the user's group becomes the group associated with the destination-file.

With the p option, cp attempts to set the owner, group, permissions, access date, and modification date to match those of the source-file.

Unlike ln (page 772), the destination-file that cp creates is independent of its source-file.

Under Mac OS X version 10.4 and later, the cp utility copies extended attributes (page 93). Under version 10.3 and earlier, cp does not copy extended attributes. The CpMac utility, provided with the Developer Tools package (page 478) and available in the /Developer/Tools directory, copies extended attributes and is available on older versions of OS X.

Examples

The first command makes a copy of the file letter in the working directory. The name of the copy is letter.sav.

$ cp letter letter.sav

The next command copies all files with filenames ending in .c into the archives directory, which is a subdirectory of the working directory. Each copied file retains its simple filename but has a new absolute pathname. Because of the p option, the copied files in archives have the same owner, group, permissions, access date, and modification date as the source files.

$ cp -p* .c archives

The next example copies memo from /Users/jenny to the working directory:

$ cp /Users/jenny/memo .

The following command copies the files named memo and letter into another directory. The copies have the same simple filenames as the source files (memo and letter) but different absolute pathnames. The absolute pathnames of the copied files are /Users/jenny/memo and /Users/jenny/letter.

$ cp memo letter /Users/jenny

The final command demonstrates one use of the f option. Alex owns the working directory and tries unsuccessfully to copy one onto a file (me) that he does not have write permission for. Because he has write permission to the directory that holds me, Alex can remove the file but not write to it. The f option enables him to overwrite one with me.

$ ls -ld
drwxrwxr-x   2 alex alex  4096 Oct 21 22:55 .
$ ls -l
-rw-r--r--   1 root admin 3555 Oct 21 22:54 me
-rw-rw-r--   1 alex alex  1222 Oct 21 22:55 one
$ cp one me
cp: cannot create regular file 'me': Permission denied
$ cp -f one me
$ ls -l
-rw-r--r--   1 alex alex  1222 Oct 21 22:58 me
-rw-rw-r--   1 alex alex  1222 Oct 21 22:55 one

Refer to "Directory Access Permissions" on page 90 for more information.

cpio: Creates an archive, restores files from an archive, or copies a directory

cpio i [options] [pattern-list]

cpio o [options]

cpio p [options] destination-directory

The cpio utility has three modes of operation: Create mode places multiple files into a single archive file, Extract mode restores files from an archive, and Copy mode makes a copy of a directory hierarchy. The archive file used by cpio may be saved on disk, removable media, or a remote system.

Create mode (o for copy-out) reads a list of ordinary or directory filenames from standard input and writes the resulting archive file to standard output. Extract mode (i for copy-in) reads an archive from standard input and extracts files from that archive. Copy mode (p for pass-through) reads a list of ordinary or directory filenames from standard input and copies the files to a directory.

Arguments

In Extract mode cpio extracts all files found in the archive that it reads from standard input. You can choose to extract or display the names of files selectively by supplying a pattern-list. If the name of a file in the archive matches a pattern in the pattern-list, that file is extracted; otherwise, it is ignored. The cpio patterns are similar to shell wildcards (page 133) except that they match slashes (/) and a leading period (.) in a filename. See the fnmatch man page for more information on cpio patterns.

In Copy mode the destination-directory is the name of the directory that cpio copies files into.

Options

A single, mandatory major option specifies the mode in which cpio operates: Create, Extract, or Copy. You can omit the other options.

Major Options

You must specify exactly one of these options.

i

(copy in) Puts cpio in Extract mode, in which it reads the archive from standard input and extracts files. Without any patterns on the command line, cpio extracts all the files from the archive. With patterns, cpio exTRacts only files with names that the patterns match. The following example extracts from the root.cpio archive file on an external drive only those files whose names end in .c:

$ cpio -i \*.c < /Volumes/Backups/root.cpio

The backslash prevents the shell from expanding the * before it passes the argument to cpio.

o

(copy out) Puts cpio in Create mode, in which it constructs an archive from the files named on standard input. These files may be ordinary or directory files, and each must appear on a separate line. The archive is written to standard output. The find utility frequently generates the filenames that cpio uses. The following command builds an archive of the startup volume (root filesystem) and writes it to the archive on /Volumes/Backups:

[View full width]
$ find -x -d / -print  cpio -o >/Volumes/Backups
  /root.cpio

The x option causes find to traverse only the startup volume; otherwise, find would eventually reach the root.cpio file and cause cpio to try to back it up. The d option causes find to search for files in a depth-first manner, reducing the likelihood of permissions problems when you restore the files from the archive. See the discussion of this option on page 696.

p

(pass through) Puts cpio in Copy mode, in which it copies files named on standard input from one place on the system to another. Instead of constructing an archive file containing the files named on standard input, cpio copies them into the destination-directory (the last argument on the cpio command line). The effect is the same as if you had created an archive in Create mode and then extracted the files in Extract mode, except that using Copy mode avoids creating an archive. The following example copies the working directory hierarchy into /Users/alex/code. Make sure that ~alex/code is not a subdirectory of the working directory or you will perform a recursive copy.

$ find -d . -print | cpio -pdm ~alex/code

Other Options

The remaining options alter the behavior of cpio. These options work with one or more of the three major options.

a | (access time) Resets the access times of source files after copying them so that they have the same access time after copying as they did before copying.
B | (block) Sets the block size to 5,120 bytes instead of the default 512 bytes.
c | (compatible) Writes header information in ASCII so that older (incompatible) cpio utilities on other systems can read the file. This option is rarely needed.
d | (directories) Creates directories as needed when copying files. For example, you need this option when you are extracting files from an archive with a file list generated by find with the d option. This option works in Extract and Copy modes only.
E | filename
 | Reads pattern-list from filename, one pattern per line. You can specify additional patterns on the command line.
f | (flip) Reverses the sense of the test done on pattern-list when extracting files from an archive. Files are extracted from the archive only if they do not match any patterns in pattern-list.
F | archive
 | (file) Uses archive as the name of the archive file. In Extract mode, reads from archive instead of standard input. In Create mode, writes to archive instead of standard output. You can use this option to access a device on another system on a network.
l | (link) When possible, links files instead of copying them.
L | (dereference) Copies the files that symbolic links point to, not the symbolic links themselves.
m | (modification) Preserves the modification times of files that are extracted from an archive. Without this option, the files show the time they were extracted. With this option, the created files show the time they had when they were copied into the archive.
r | (rename) Allows you to rename files as cpio copies them. When cpio prompts you with the name of a file, you can respond with the new name. The file is then copied with the new name. If you press RETURN without entering a name, cpio does not copy the file.
t | (table of contents) Displays a table of contents of the archive. This option works only with the i option, although no files are extracted from the archive. With the v option, it displays a detailed table of contents in a format similar to that displayed by ls l.
u | (unconditional) Overwrites existing files regardless of their modification times. Without this option, cpio will not overwrite a more recently modified file with an older one; it simply displays a warning message.
v | (verbose) Lists files as they are processed. With the t option, it displays a detailed table of contents in a format similar to that displayed by ls l.

Discussion

Without the u option, cpio will not overwrite a more recently modified file with an older file.

You can use both ordinary and directory filenames as input when you create an archive. If the name of an ordinary file appears in the input list before the name of its parent directory, the ordinary file appears before its parent directory in the archive as well. This order can lead to an avoidable error: When you extract files from the archive, the child has nowhere to go in the file structure if its parent has not yet been extracted.

Making sure that files appear after their parent directories in the archive is not always a solution. Consider the problem that occurs if the m option is used when extracting files. Because the modification time of a parent directory is updated whenever a file is created within it, the original modification time of the parent directory is lost when the first file is written to it.

The solution to this potential problem is to ensure that all files appear before their parent directories when creating an archive and to create directories as needed when extracting files from an archive. When you use this technique, directories are extracted only after all files have been written to them and their modification times are preserved.

With the d option, the find utility generates a list of files in which all children appear in the list before their parent directories. If you use this list to create an archive, the files are in the proper order. (Refer to the first example in "Examples.") When extracting files from an archive, the d option causes cpio to create parent directories as needed and the m option preserves modification times. Using this combination of utilities and options preserves directory modification times through a create/extract sequence.

This strategy also solves another potential problem. Sometimes a parent directory may not have permissions set so that you can extract files into it. When cpio automatically creates the directory with d, you can be assured that you have write permission to the directory. When the directory is extracted from the archive (after all the files are written into the directory), it is extracted with its original permissions.

Notes

Under Mac OS X version 10.4 and later, cpio copies extended attributes (page 93). Under version 10.3 and earlier, cpio does not copy extended attributes. For a similar utility that does copy extended attributes, see ditto (page 715).

Examples

The first example creates an archive of the files in Jenny's home directory, writing the archive file to a backup disk:

$ find -d /Users/jenny -print | cpio -o >/Volumes/Backups/jenny.cpio

The find utility produces the filenames that cpio uses to build the archive. The d option causes all entries in a directory to be listed before the directory name itself, making it possible for cpio to preserve the original modification times of directories (see the preceding "Discussion"). Use the d and the m options when you extract files from this archive (see the following examples).

The next example displays a detailed listing of the files in the archive file:

$ cpio -itv < /Volumes/Backups/jenny.cpio

The following command restores the files that formerly were in the memo subdirectory of Jenny's home directory:

$ cpio -idm /Users/jenny/memo/\* < /Volumes/Backups/jenny.cpio

The d option ensures that any subdirectories that were in the memo directory are re-created as needed. The m option preserves the modification times of files and directories. The asterisk in the regular expression is escaped to keep the shell from expanding it.

The final example uses the f option to restore all files in the archive except those that were formerly in the memo subdirectory:

$ cpio -ivmdf /Users/jenny/memo/\* < /Volumes/Backups/jenny.cpio

The v option lists the extracted files as cpio processes the archive, verifying that the expected files are extracted.

crontab: Maintains crontab files

crontab [u username] filename

crontab [u username] option

A crontab file associates periodic times (such as 14:00 on Wednesdays) with commands. The cron utility executes each command at the specified time. When you are working as yourself, the crontab utility installs, removes, lists, and edits your own crontab file. Superuser can work with any user's crontab file.

Arguments

The first format copies the contents of filename (which contains crontab commands) into the crontab file of the user who runs the command. When the user does not have a crontab file, this process creates a new one; when the user already has a crontab file, it overwrites the file. When you replace filename with a hyphen (-), crontab reads from standard input.

The second format lists, removes, or edits the crontab file, depending on which option you specify.

Options

Choose only one of the e, l, and r options. Superuser can use u with one of these options.

e | (edit) Runs the text editor specified by the VISUAL or EDITOR shell variable on the crontab file, enabling you to add, change, or delete entries. Installs the modified crontab file when you exit from the editor.
l | (list) Displays the contents of the crontab file.
r | (remove) Deletes the crontab file.
u | username
 | (user) Works on the crontab file belonging to username. Only Superuser can use this option, and Superuser should always use this option.

Notes

This section covers the versions of cron, crond, crontab, and crontab files that were written by Paul Vixiehence the term Vixie cron. These versions differ from an earlier version of Vixie cron as well as from the classic SVR3 syntax. This version is POSIX compliant. Mac OS X 10.4 and later use cron for user jobs only; system jobs are handled by launchd (page 456).

User crontab files are kept in the /var/cron/tabs directory. Each of these files is named with the username of the user that it belongs to, owned by root, and associated with the user's primary group.

The system utility named cron reads the crontab files and runs the commands. If a command line in a crontab file does not redirect its output, output sent to standard output and standard error is mailed to the user unless you set the MAILTO variable within the crontab file to a different username.

Following is a typical user crontab file:

$ crontab -l
07 4 * * *      /Users/alex/bin/update_mirrors
13 1 * * *      /Users/alex/weblog/nightly

Each entry in a crontab file begins with five fields that specify when the command is to run (minute, hour, day of the month, month, and day of the week). The cron utility interprets an asterisk appearing in place of a number as a wildcard representing all possible values. In the day-of-the-week field, you can use either 7 or 0 to represent Sunday. The preceding example runs update_mirrors daily at 4:07 AM and nightly daily at 1:13 AM.

It is a good practice to start cron jobs a variable number of minutes before or after the hour, half hour, or quarter hour. When you start them at these times, it becomes less likely that many processes will start at the same time, potentially overloading the system.

The cron daemon is started by launchd. If cron has no jobs to run, it exits. It starts automatically when cron jobs are created. When cron starts, it reads all crontab files into memory. The cron utility mostly sleeps but wakes up once a minute, reviews all crontab entries it has stored in memory, and runs whichever ones are due to be run at that time.

/var/cron/allow/var/cron/deny

Superuser determines which users can run crontab by creating, editing, and removing the /var/cron/allow and /var/cron/deny files. When you create a deny file with no entries and no allow file exists, everyone can use crontab. When the allow file exists, only users listed in that file can use crontab, regardless of the presence and contents of deny. Otherwise, you can list in the allow file those users who are allowed to use crontab and in the deny file those who are not allowed to use it. (Listing a user in deny is not strictly necessary because, if an allow file exists and the user is not listed in it, the user will not be able to use crontab anyway.)

Examples

The following example uses crontab l to list the contents of Jenny's crontab file (/var/cron/tabs/jenny). All the scripts that Jenny runs are in her ~/bin directory. The first line sets the MAILTO variable to alex so that Alex gets the output from commands run from Jenny's crontab file that is not redirected. The sat.job script runs every Saturday (day 6) at 2:05 AM, twice.week runs at 12:02 AM on Sunday and Thursday (days 0 and 4), and twice.day runs twice a day, every day, at 10:05 AM and 4:05 PM.

$ who am i
jenny

$ crontab -l
MAILTO=alex
05 02 * * 6        $HOME/bin/sat.job
00 02 * * 0,4      $HOME/bin/twice.week
05 10,16 * * *     $HOME/bin/twice.day

To add an entry to your crontab file, run the crontab utility with the e (edit) option.

cut: Selects characters or fields from input lines

cut [options] [file-list]

The cut utility selects characters or fields from lines of input and writes them to standard output. Character and field numbering start with 1.

Arguments

The file-list specifies the pathnames of one or more ordinary files that cut will work with. If you do not specify an argument or if you specify a hyphen (-) in place of a filename, cut reads from standard input.

Options

c | clist
 | (characters) Selects the characters given by the column numbers in clist. The value of clist is one or more comma-separated column numbers or column ranges. A range is specified by two column numbers separated by a hyphen. A range of -n means columns 1 through n; n means columns n through the end of the line.
d | dchar
 | (delimiter) Specifies dchar as the input and output field delimiter. The default delimiter is a TAB character. Quote characters as necessary to protect them from shell expansion.
f | flist
 | (fields) Selects the fields specified in flist. The value of flist is one or more comma-separated field numbers or field ranges. A range is specified by two field numbers separated by a hyphen. A range of -n means fields 1 through n; n- means fields n through the last field. The field delimiter is a TAB character unless you use the -d option to change it.

Notes

Although limited in functionality, cut is easy to learn and use and is a good choice when columns and fields can be selected without using pattern matching. Sometimes cut is used with paste (page 807).

Examples

For the next two examples, assume that an ls l command produces the following output:

$ ls -l
total 2944
-rwxr-xr-x 1 zach zach     259 Feb  1 00:12 countout
-rw-rw-r-- 1 zach zach    9453 Feb  4 23:17 headers
-rw-rw-r-- 1 zach zach 1474828 Jan 14 14:15 memo
-rw-rw-r-- 1 zach zach 1474828 Jan 14 14:33 memos_save
-rw-rw-r-- 1 zach zach    7134 Feb 4  23:18 tmp1
-rw-rw-r-- 1 zach zach    4770 Feb 4  23:26 tmp2
-rw-rw-r-- 1 zach zach   13580 Nov 7  08:01 typescript

The following command outputs the permissions of the files in the working directory. The cut utility with the c option selects characters 2 through 10 from each input line. The characters in this range are written to standard output.

$ ls -l | cut -c2-10
otal 2944
rwxr-xr-x
rw-rw-r--
rw-rw-r--
rw-rw-r--
rw-rw-r--
rw-rw-r--
rw-rw-r--

The next command outputs the size and name of each file in the working directory. This time the f option selects the fifth and ninth fields from the input lines. The d option tells cut to use SPACEs, not TABs, as delimiters. The tr utility (page 879) with the s option changes sequences of more than one SPACE character into a single SPACE; otherwise, cut counts the extra SPACE characters as separate fields.

$ ls -l | tr -s '  ' ' ' | cut -f5,9 -d' '
259 countout
9453 headers
1474828 memo
1474828 memos_save
7134 tmp1
4770 tmp2
13580 typescript

The last example displays a list of full names that are stored in the eighth field of the passwd database. The nidump utility (page 796) sends its output to standard output, and a pipe redirects the output to standard input of cut. The d option specifies that the colon character be used as the field delimiter.

$ nidump passwd /
nobody:*:-2:-2::0:0:Unprivileged User:/var/empty:/usr/bin/false
root:********:0:0::0:0:System Administrator:/private/var/root:/bin/sh
...
alex:********:501:501::0:0:Alex Watson:/Users/alex:/bin/bash
jenny:*:502:502::0:0:Jenny Chen:/Users/jenny:/bin/zsh
...

$ nidump passwd / | cut -d: -f8 -
Unprivileged User
System Administrator
...
Alex Watson
Jenny Chen

date: Displays or sets the system time and date

date [options] [+format]

date [options] [newdate]

The date utility displays the time and date known to the system. Superuser can use date to change the system clock.

Arguments

The +format argument specifies the format for the output of date. The format string consisting of field descriptors and text follows a plus sign (+). The field descriptors are preceded by percent signs, and each one is replaced by its value in the output. Table VI-9 lists some of the field descriptors.

Table VI-9. Field descriptors

Descriptor

Meaning

%a

Abbreviated weekdaySun to Sat

%A

Unabbreviated weekdaySunday to Saturday

%b

Abbreviated monthJan to Dec

%B

Unabbreviated monthJanuary to December

%c

Date and time in default format used by date

%d

Day of the month01 to 31

%D

Date in mm/dd/yy format

%H

Hour00 to 23

%I

Hour00 to 12

%j

Julian date (day of the year001 to 366)

%m

Month of the year01 to 12

%M

Minutes00 to 59

%n

NEWLINE character

%p

AM or PM

%r

Time in AM/PM notation

%s

Number of seconds since the beginning of January 1, 1970

%S

Seconds00 to 60 (the 60 accommodates leap seconds)

%t

TAB character

%T

Time in HH:MM:SS format

%w

Day of the week0 to 6 (0 = Sunday)

%y

Last two digits of the year00 to 99

%Y

Year in four-digit format (for example, 2005)

%Z

Time zone (for example, PDT)

The date utility zero fills numeric fields.

The date utility assumes that, in a format string, characters that are not percent signs or field descriptors are ordinary text; date copies these characters to standard output. You can use ordinary text to add punctuation to the date and to add labels (for example, you can put the word DATE: in front of the date). Surround the format argument with single quotation marks if it contains SPACEs or other characters that have a special meaning to the shell.

Setting the date

When Superuser specifies newdate, the system changes the system clock to reflect the new date. The newdate argument has the format

nnddhhmm[[cc]yy][.ss]

where nn is the number of the month (0112), dd is the day of the month (0131), hh is the hour based on a 24-hour clock (0023), and mm is the minutes (0059). When you change the date, you must specify at least these fields.

The optional cc specifies the first two digits of the year (the value of the century minus 1), and yy specifies the last two digits of the year. You can specify yy or ccyy following mm. When you do not specify a year, date assumes that the year has not changed.

You can specify the number of seconds past the start of the minute with .ss.

Options

r | seconds
 | Displays the date and time that is seconds seconds after the epoch (midnight, January 1, 1970 UTC).
u | (UTC) Displays or sets the time and date using Universal Coordinated Time (UTC, page 960; also called Greenwich Mean Time [GMT]).

Notes

If you set up a locale database, date uses that database to substitute terms appropriate to your locale (page 941).

Examples

The first example shows how to set the date for 2:07:30 PM on August 19 without changing the year:

# date 08191407.30
Fri Aug 19 14:07:30 PDT 2005

The next example shows the format argument, which causes date to display the date in a commonly used format:

$ date '+Today is %h %d, %Y'
Today is Aug 19, 2005

dd: Converts and copies a file

dd [arguments]

The dd (device-to-device copy) utility converts and copies a file. The primary use of dd is to copy files to and from such devices as tape and floppy drives. Often dd can handle the transfer of information to and from other operating systems when other methods fail. A rich set of arguments gives you precise control over the characteristics of the transfer.

Arguments

Accepts the common options described on page 663.

By default dd copies standard input to standard output.

bs=n | (block size) Reads and writes n bytes at a time. This argument overrides the ibs and obs arguments.
cbs=n | (conversion block size) When performing data conversion during the copy, converts n bytes at a time.
conv=type[,type...]
 | By applying conversion types in the order given on the command line, converts the data that is being copied. The types must be separated by commas with no SPACEs. Table VI-10 lists the conversion types.

Table VI-10. Conversion types

type

Meaning

ascii

Converts EBCDIC-encoded characters to ASCII, allowing you to read tapes written on IBM mainframe and similar computers.

block

Each time a line of input (that is, a sequence of characters terminated with a NEWLINE character) is read, outputs a block of text without the NEWLINE. Each output block has the size given by the obs or bs argument and is created by adding trailing SPACE characters to the text until it reaches the proper size.

ebcdic

Converts ASCII-encoded characters to EBCDIC, allowing you to write tapes for use on IBM mainframe and similar computers.

unblock

Performs the opposite of a block conversion.

lcase

Converts uppercase letters to lowercase while copying data.

noerror

If a read error occurs, dd normally terminates. This conversion allows dd to continue processing data. This conversion is useful when you are trying to recover data from bad media.

notrunc

Does not truncate the output file before writing to it.

ucase

Converts lowercase letters to uppercase while copying data.

count=numblocks | Restricts to numblocks the number of blocks of input that dd copies. The size of each block is specified by the ibs argument.
ibs=n | (input block size) Reads n bytes at a time.
if=filename | (input file) Reads from filename instead of from standard input. You can specify a device name for filename to read from that device.
obs=n | (output block size) Writes n bytes at a time.
of=filename | (output file) Writes to filename instead of to standard output. You can specify a device name for filename to write to that device.
seek=numblocks | Skips numblocks blocks of output before writing any output. The size of each block is specified by the obs argument.
skip=numblocks | Skips numblocks blocks of input before starting to copy. The size of each block is specified by the ibs argument.

Notes

You can use some of the standard multiplicative suffixes to make it easier to specify large block sizes, but dd uses lowercase letters instead of the uppercase letters shown in Table VI-1 on page 662. In addition to the k, m, and g suffixes, dd supports b (block, multiply by 512) and w (word, multiply by the number of bytes in an integer).

Examples

The dd utility can create simple disk images. The following example makes a raw disk image of the partition disk0s1:

# dd if=/dev/disk0s1 of=backup

You can also use dd to create a file filled with pseudorandom bytes:

$ dd if=/dev/random of=randfile2 bs=1 count=100

The preceding command reads from the /dev/random file (an interface to the kernel's random number generator) and writes to the file named randfile. The block size is 1 and the count is 100, so randfile is 100 bytes long. See the random man page in section 4 of the system manual for more information.

Wiping a file

You can use a similar technique to wipe data from a file before deleting it, making it almost impossible to recover data from the deleted file. You might want to wipe a file for security reasons. Wipe a file several times for added security.

In the following example, ls shows the size of the file named secret; dd, with a block size of 1 and a count corresponding to the number of bytes in secret, then wipes the file. The conv=notrunc argument ensures that dd writes over the data in the file and not another place on the disk.

$ ls -l secret
-rw-rw-r--  1 max max 2494 Feb  6 00:56 secret
$ dd if=/dev/random of=secret bs=1 count=2494 conv=notrunc
2494+0 records in
2494+0 records out
$ rm secret

df: Displays disk space usage

df [options] [filesystem-list]

The df (disk free) utility reports on the total space and the free space on each mounted device.

Arguments

When you call df without an argument, it reports on the free space on each mounted device.

The filesystem-list is an optional list of one or more pathnames that specify the filesystems that the report is to cover. You can refer to a mounted filesystem by its device pathname or by the pathname of the directory it is mounted on.

Options

h | (human readable) Displays sizes in K (kilobyte), M (megabyte), and G (gigabyte) blocks, as appropriate.
i | (inode) Reports the number of inodes (page 937) that are used and free in addition to reporting on blocks.
k | (kilobyte) Displays usage in 1,024-byte blocks.
l | (local) Displays local filesystems only.
T | fstype
 | (type) Reports information only about the filesystems of type fstype, such as hfs or nfs. Repeat this option to report on several types of filesystems.

Notes

The df utility supports the BLOCKSIZE environment variable (page 663) and ignores block sizes smaller than 512 bytes or larger than 1 gigabyte.

The count of used and free inodes is meaningless on HFS+ filesystems, where new files can be created as long as free space is available in the filesystem.

Examples

In the following example, df displays information about all mounted filesystems on the local system:

$ df
Filesystem        512-blocks     Used     Avail Capacity Mounted on
/dev/disk0s3        64465984  34221472  29732512   54%   /
devfs                    215       215         0  100%   /dev
fdesc                      2         2         0  100%   /dev
<volfs>                 1024      1024         0  100%   /.vol
/dev/disk1s9       102400000     54600 102345400    0%   /Volumes/Spare
/dev/disk1s10      385995336 119741320 266254016   31%   /Volumes/House
/dev/disk0s5       255182704 180721088 74461616    71%    /Volumes/Users
automount -nsl [263]       0        0        0   100%    /Network
automount -fstab [271]     0       0        0   100%    /automount/Servers
automount -static [271]    0        0        0   100%   /automount/static

Next df is called with the l and h options, generating a human-readable list of local filesystems. The sizes in this listing are given in terms of megabytes and gigabytes.

$ df -lh
Filesystem          Size   Used  Avail Capacity  Mounted on
/dev/disk0s3         31G    16G    14G    54%    /
/dev/disk1s9         49G    27M    49G     0%    /Volumes/Spare
/dev/disk1s10       184G    57G   127G    31%    /Volumes/House
/dev/disk0s5        122G    86G    36G    71%    /Volumes/Users

The next example uses the BLOCKSIZE environment variable to cause df to display information about the /Volumes/House partition in megabyte units, then in 1,000,000-byte units:

$ export BLOCKSIZE
$ BLOCKSIZE=1M
$ df /Volumes/House
Filesystem    1M-blocks  Used  Avail Capacity  Mounted on
/dev/disk1s10    188474 58467 130006    31%    /Volumes/House
$ BLOCKSIZE=1000000
$ df /Volumes/House
Filesystem    1000000-blocks  Used  Avail Capacity  Mounted on
/dev/disk1s10         197743 61342 136400    31%    /Volumes/House

diff: Displays the differences between two files

diff [options] file1 file2

diff [options] file1 directory

diff [options] directory file2

diff [options] directory1 directory2

The diff utility displays line-by-line differences between two files. By default diff displays the differences as instructions you can use to edit one of the files to make it the same as the other.

Arguments

The file1 and file2 are pathnames of ordinary files that diff works on. When the directory argument is used in place of file2, diff looks for a file in directory with the same name as file1. It works similarly when directory replaces file1. When you specify two directory arguments, diff compares the files in directory1 with the files that have the same simple filenames in directory2.

Options

Accepts the common options described on page 663, with one exception: When one of the arguments is a directory and the other is an ordinary file, you cannot compare to standard input.

ignorespace-change |
b | Ignores whitespace (SPACEs and TABs) at the ends of lines and considers other strings of whitespace to be equal.
ignoreblanklines |
B | Ignores differences that involve only blank lines.
context[=lines] C | [lines]
 | Displays the sections of the two files that differ, including lines lines (the default is 3) around each line that differs to show the context. Each line in file1 that is missing from file2 is preceded by ; each extra line in file2 is preceded by +; and lines that are different in the two files are marked with !. When lines that differ are within lines lines of each other, they are grouped together in the output.
ed e | Creates and sends to standard output a script for the ed editor, which will edit file1 to make it the same as file2. You must add w (write) and q (quit) instructions to the end of the script if you plan to redirect input to ed from the script. When you use ed, diff displays the changes in reverse order: Changes to the end of the file are listed before changes to the top, preventing early changes from affecting later changes when the script is used as input to ed. For example, if a line near the top were deleted, subsequent line numbers in the script would be wrong.
ignore-case i | Ignores differences in case when comparing files.
new-file N | When comparing directories, when a file is present in one of the directories only, considers it to be present and empty in the other directory.
show-c-function |
p | Shows which C function each change affects.
brief q | Does not display the differences between lines in the files. Instead, diff reports only that the files differ.
recursive r | When using diff to compare the files in two directories, causes the comparisons to extend through subdirectories as well.
unified[=lines] U | lines
 | Uses the easier-to-read unified output format. See the discussion of diff on page 49 for more detail and an example. The lines is the number of lines of context; the default is 3.
ignore-all-space |
w | (whitespace) Ignores whitespace when comparing lines.
width=columns |
W | columns
 | Sets the width of the columns that diff uses to display the output to columns. This option is useful with the side-by-side option. The sdiff utility uses a lowercase w to perform the same function: w columns.
side-by-side y | Displays output in a side-by-side format. This option generates the same output as sdiff. Use the width=columns option with this option.

Discussion

When you use diff without any options, it produces a series of lines containing Add (a), Delete (d), and Change (c) instructions. Each of these lines is followed by the lines from the file that you need to add, delete, or change to make the files the same. A less than symbol (<) precedes lines from file1. A greater than symbol (>) precedes lines from file2. The diff output appears in the format shown in Table VI-11. A pair of line numbers separated by a comma represents a range of lines; a single line number represents a single line.

Table VI-11. diff output

Instruction

Meaning (to change file1 to file2)

line1 a line2,line3
> lines from file2

Append line2 through line3 from file2 after line1 in file1

line1,line2 d line3
< lines from file1

Delete line1 through line2 from file1

line1,line2 c line3,line4
< lines from file1

> lines from file 2

Change line1 through line2 in file1 to line3 through line 4 from file2

The diff utility assumes that you will convert file1 to file2. The line numbers to the left of each of the a, c, or d instructions always pertain to file1; the line numbers to the right of the instructions apply to file2. To convert file2 to file1, run diff again, reversing the order of the arguments.

Notes

The sdiff utility is similar to diff but its output may be easier to read. The diff side-by-side option produces the same output as sdiff. See the "Examples" section and refer to the diff and sdiff man and info pages for more information.

The diff3 utility compares three files.

Examples

The first example shows how diff displays the differences between two short, similar files:

$ cat m
aaaaa
bbbbb
ccccc

$ cat n
aaaaa
ccccc

$ diff m n
2d1
< bbbbb

The difference between files m and n is that the second line of file m (bbbbb) is missing from file n. The first line that diff displays (2d1) indicates that you need to delete the second line from file1 (m) to make it the same as file2 (n). The next line diff displays starts with a less than symbol (<), indicating that this line of text comes from file1. In this example, you do not need this informationall you need to know is the line number so that you can delete the line.

The side-by-side option and the sdiff utility, both with the output width set to 30 columns (characters), display the same output. In the output a less than symbol points to the extra line in file m, whereas diff/sdiff leaves a blank line in file n where the extra line would go to make the files the same.

$ diff --side-by-side --width=30 m n
aaaaa           aaaaa
bbbbb         <
ccccc           ccccc

$ sdiff -w 30 m n
aaaaa           aaaaa
bbbbb         <
ccccc           ccccc

The next example uses the same m file and a new file, p, to show diff issuing an a (append) instruction:

$ cat p
aaaaa
bbbbb
rrrrr
ccccc
$ diff m p
2a3
> rrrrr

In the preceding example, diff issues the instruction 2a3 to indicate that you must append a line to file m after line 2 to make it the same as file p. The second line that diff displays indicates that the line comes from file p (the line begins with >, indicating file2). In this example, you need the information on this line; the appended line must contain the text rrrrr.

The next example uses file m again, this time with file r, to show how diff indicates a line that needs to be changed:

$ cat r
aaaaa
-q
ccccc

$ diff m r
2c2
< bbbbb
---
> q

The difference between the two files occurs in line 2: File m contains bbbbb, and file r contains q. The diff utility displays 2c2 to indicate that you need to change line 2. Next diff shows that you must change line 2 in file m (bbbbb) to line 2 in file r (q) to make the files the same. The three hyphens indicate the end of the text in file m that needs to be changed and the start of the text in file r that is to replace it.

Comparing the same files using the side-by-side and width options (y and W) yields an easier-to-read result. The pipe symbol (|) indicates that the line on one side must replace the line on the other side to make the files the same:

$ diff -y -W 30 m r
aaaaa           aaaaa
bbbbb         | -q
ccccc           ccccc

The next examples compare the two files q and v:

$ cat q        $ cat v
Monday         Monday
Tuesday        Wednesday
Wednesday      Thursday
Thursday       Thursday
Saturday       Friday
Sunday         Saturday
               Sundae

Running in side-by-side mode diff shows that Tuesday is missing from file v, there is only one Thursday in file q (compared with two in file v), and Friday is missing from file q. The last line is Sunday in file q and Sundae in file v: diff indicates that these lines are different. You can change file q to be the same as file v by removing Tuesday, adding one Thursday and Friday, and substituting Sundae from file v for Sunday from file q. Alternatively, you can change file v to be the same as file q by adding Tuesday, removing one Thursday and Friday, and substituting Sunday from file q for Sundae from file v.

$ diff -y -W 30 q v
Monday          Monday
Tuesday       <
Wednesday       Wednesday
Thursday        Thursday
              > Thursday
              > Friday
Saturday        Saturday
Sunday        | Sundae

Context diff

With the context option (called a context diff), diff displays output that tells you how to turn the first file into the second file. The top two lines identify the files and show that q is represented by asterisks, whereas v is represented by hyphens. A row of asterisks that indicates the start of a hunk of text is followed by a row of asterisks with the numbers 1,6 in the middle. This line indicates that the instructions in the first section tell you what to remove from or change in file qnamely, lines 1 through 6 (that is, all the lines of file q; in a longer file it would mark the first hunk). The hyphen on the next line means that you need to remove the line with Tuesday. The line with an exclamation point indicates that you need to replace the line with Sunday with the corresponding line from file v. The row of hyphens with the numbers 1,7 in the middle indicates that the next section tells you which lines from file vlines 1 through 7you need to add or change in file q. You need to add a second line with Thursday and a line with Friday, and you need to change Sunday in file q to Sundae (from file v).

$ diff --context q v
***  q   Mon Aug 22 18:26:45 2005
--- v   Mon Aug 22 18:27:55 2005
***************
*** 1,6 ****
  Monday
- Tuesday
  Wednesday
  Thursday
  Saturday
! Sunday
--- 1,7 ----
  Monday
  Wednesday
  Thursday
+ Thursday
+ Friday
  Saturday
! Sundae

diskutil: Checks, modifies, and repairs local volumes

diskutil action [arguments]

The diskutil utility mounts, unmounts, and displays information about disks and partitions (volumes). It can also format and repair filesystems and divide a disk into partitions.

Arguments

The action specifies what diskutil is to do. Table VI-12 lists common actions along with the argument each takes.

Table VI-12. diskutil actions

Action

Argument

Description

eraseVolume

type name device

Reformats device using the format type and the label name.

The filesystem type is most typically HFS+, but can also be UFS or MS-DOS.

The name specifies the name of the volume; alphanumeric names are the easiest to work with. You can specify additional options as part of the type. For example, a FAT32 filesystem (as used in Windows 98 and later) would have a type of MS-DOS FAT32. A journaled, case-sensitive, HFS+ filesystem would have a type of Case-sensitive Journaled HFS+.

info

device

Displays information about device. Does not require ownership of device.

list

[device]

Lists partitions on device. Without device lists partitions on all devices. Does not require ownership of device.

mount

device

Mounts device.

mountDisk

device

Mounts all devices on the disk containing device.

reformat

device

Reformats device using its current name and format.

repairVolume

device

Repairs the filesystem on device.

unmount

device

Unmounts device.

unmountDisk

device

Unmounts all devices on the disk containing device.

verifyVolume

device

Verifies the filesystem on device. Does not require ownership of device.

Notes

The diskutil utility provides access to the Disk Management framework, the support code used by the Disk Utility application. It allows some choices that are not supported from the graphical interface. For example, the Disk Utility application on non-Server systems does not support the HFS+ Extended (case-sensitive) format, but diskutil can format a disk as an HFS+ Extended filesystem on any version of Mac OS X.

You must own device, or be working with root privileges, when you specify an action that modifies or changes the state of a volume.

fsck

The diskutil verifyVolume and repairVolume actions are analogous to the fsck utility on other UNIX systems. Under OS X the fsck utility is deprecated except when the system is in single-user mode (page 437).

disktool

Some of the functions performed by diskutil were handled by disktool in the past.

Examples

The first example displays a list of disk devices and volumes available on the local system:

$ diskutil list
/dev/disk0
   #:                   type name                 size      identifier
   0: Apple_partition_scheme                      *152.7 GB disk0
   1:    Apple_partition_map                      31.5 KB   disk0s1
   2:              Apple_HFS Eva01                30.7 GB   disk0s3
   3:              Apple_HFS Users                121.7 GB  disk0s5
/dev/disk1
   #:                   type name                 size      identifier
   0: Apple_partition_scheme                      *232.9 GB disk1
   1:    Apple_partition_map                      31.5 KB   disk1s1
   2:         Apple_Driver43                      28.0 KB   disk1s2
   3:         Apple_Driver43                      28.0 KB   disk1s3
   4:       Apple_Driver_ATA                      28.0 KB   disk1s4
   5:       Apple_Driver_ATA                      28.0 KB   disk1s5
   6:         Apple_FWDriver                      256.0 KB  disk1s6
   7:     Apple_Driver_IOKit                      256.0 KB  disk1s7
   8:          Apple_Patches                      256.0 KB  disk1s8
   9:              Apple_HFS Spare                48.8 GB   disk1s9
  10:              Apple_HFS House                184.1 GB  disk1s10

The next example displays information about one of the mounted volumes:

$ diskutil info disk1s9
   Device Node:        /dev/disk1s9
   Device Identifier:  disk1s9
   Mount Point:        /Volumes/Spare
   Volume Name:        Spare

   File System:        HFS+
   Owners:             Enabled
   Partition Type:     Apple_HFS
   Bootable:           Is bootable
   Media Type:         Generic
   Protocol:           FireWire
   SMART Status:       Not Supported
   UUID:               C77BB3DC-EFBB-30B0-B191-DE7E01D8A563

   Total Size:         48.8 GB
   Free Space:         48.8 GB

   Read Only:          No
   Ejectable:          Yes

The next example formats the partition at /dev/disk1s8 as an HFS+ Extended (HFSX) filesystem and labels it Spare2. This command erases all data on the partition:

# diskutil eraseVolume 'Case-sensitive HFS+' Spare2 disk1s10
Started erase on disk disk1s10
Erasing
Mounting Disk
Finished erase on disk disk1s10

The final example shows the output of a successful verifyVolume operation:

$ diskutil verifyVolume disk1s9
Started verify/repair on volume disk1s9 Spare
Checking HFS Plus volume.
Checking Extents Overflow file.
Checking Catalog file.
Checking Catalog hierarchy.
Checking volume bitmap.
Checking volume information.
The volume Spare appears to be OK.
Mounting Disk
Verify/repair finished on volume disk1s9 Spare

ditto: Copies files and creates and unpacks archives

ditto [options] source-file destination-file

ditto [options] source-file-list destination-directory

ditto c [options] source-directory destination-archive

ditto x [options] source-archive-list destination-directory

The ditto utility copies files, including extended attributes (page 93). It can copy to and from cpio and zip archive files, as well as copy ordinary files and directories.

Arguments

The source-file is the pathname of the file that ditto is to make a copy of. The destination-file is the pathname that ditto assigns to the resulting copy of the file.

The source-file-list specifies one or more pathnames of files and/or directories that ditto makes copies of. The destination-directory is the pathname of the directory that ditto copies the files and directories into. When you specify a destination-directory, ditto gives each of the copied files the same simple filename as its source-file.

The source-directory is a single directory that ditto copies into the destination-archive. The resulting archive holds copies of the contents of source-directory, but not the directory itself.

The source-archive-list specifies one or more pathnames of archives that ditto extracts into destination-directory.

Using a hyphen (-) in place of a filename or directory name causes ditto to read from standard input or write to standard output instead of reading from or writing to that file or directory.

Options

You cannot use the c and x options together.

c | (create archive) Creates an archive file.
x | (extract archive) Extracts files from an archive file.
help | Displays a help message.
k | (pkzip) Uses the zip format, instead of the default cpio (page 693) format, to create or extract archives. For more information on zip, see the tip named "gzip versus zip" on page 56.
norsrc | (no resource) Ignores extended attributes. This option causes ditto to copy only data forks (the default behavior under Mac OS X 10.3 and earlier).
rsrc | (resource) Copies extended attributes, including resource forks (the default behavior under Mac OS X 10.4 and later). Also, rsrc and rsrcFork.
v | (verbose) Sends a line to standard error for each directory copied.
V | (very verbose) Sends a line to standard error for each file, symbolic link, and device node copied.
X | (exclude) Prevents ditto from searching directories in filesystems other than the filesystem(s) that hold the files it was explicitly told to copy.
z | (compress) Uses gzip (page 756) or gunzip to compress or decompress cpio archives.

Notes

The ditto utility does not copy the locked attribute flag (page 95). See the tip on page 92 regarding the uchg flag. The utility also does not copy ACLs.

By default ditto creates and reads archives (page 921) in the cpio (page 693) format.

The ditto utility cannot list the contents of archive files. It can only create or extract files from archives. Use pax or cpio to list the contents of cpio archives, and use unzip with the l option to list the contents of zip files.

Examples

The following examples show three ways to back up a user's home directory, including extended attributes (except as mentioned in "Notes"), preserving timestamps and permissions. For these examples, include the rsrc option if you are working under OS X 10.3 or earlier. The first example copies Zach's home directory to the volume (filesystem) named Backups; the copy is a new directory named zach.0228:

$ ditto /Users/zach /Volumes/Backups/zach.0228

The next example copies Zach's home directory into a single cpio-format archive file on the volume named Backups:

$ ditto -c /Users/zach /Volumes/Backups/zach.0228.cpio

The next example copies Zach's home directory into a zip archive:

$ ditto -c -k /Users/zach /Volumes/Backups/zach.0228.zip

Each of the next three examples restores the corresponding backup archive into Zach's home directory, overwriting any files that are already there:

$ ditto /Volumes/Backups/zach.0228 /Users/zach
$ ditto -x /Volumes/Backups/zach.0228.cpio /Users/zach
$ ditto -x -k /Volumes/Backups/zach.0228.zip /Users/zach

You can use a hyphen (-) in place of a filename or directory name to read from standard input or write to standard output in place of reading from or writing to that file or directory. The following example copies the Scripts directory to a directory named ScriptsBackups on the remote host bravo. It uses an argument of a hyphen in place of source-directory locally to write to standard output and in place of destination-directory on the remote system to read from standard input:

$ ditto -c Scripts - | ssh bravo ditto -x - ScriptsBackups

The final example copies the local startup disk (the root filesystem) to the volume named Backups.root. Because some of the files can be read only as root, the script must be run by a user with root privileges. The X option keeps ditto from trying to copy other volumes (filesystems) that are mounted under /.

# ditto -X / /Volumes/Backups.root

dmesg: Displays kernel messages

dmesg [options]

The dmesg utility displays the kernel ring buffer, the buffer that the kernel stores its messages in.

Options

M | core
 | The core is the name of the file to process (defaults to /dev/kmem). The core file should be a core dump from an OS X kernel.
N | kernel
 | The kernel is the pathname of a kernel file (defaults to /mach). If you are displaying information about a core dump, kernel should be the kernel that was running at the time the core file was created.

Discussion

When the system boots, the kernel fills its ring buffer with messages regarding hardware and module initialization. Messages in the kernel ring buffer are often useful for diagnosing system problems.

Notes

As a ring buffer, the kernel message buffer keeps the most recent messages it receives, discarding the oldest messages once it fills up. To save a list of kernel boot messages, give the following command immediately after booting the system and logging in:

$ dmesg > dmesg.boot

This command saves the kernel messages in the dmesg.boot file. This list can be educational and quite useful when you are having a problem with the boot process.

The dmesg utility displays the same messages that the system displays on the text console when you hold down COMMAND-V while booting (verbose boot). The /var/log/system.log file holds manybut not allof the kernel messages.

Examples

The following command displays kernel messages in the ring buffer with the string serial in them, regardless of case. It shows that the kernel found a serial port device.

$ dmesg | grep -i serial
Apple16X50PCI2: Identified 4 Serial channels at PCI SLOT-2 Bus=5 Dev=2 Func=0

du: Displays information on disk usage by file

du [options] [path-list]

The du (disk usage) utility reports how much disk space is occupied by a directory hierarchy or a file. By default du displays the number of 1,024-byte blocks occupied by the directory or file.

Arguments

Without any arguments, du displays information about the working directory and its subdirectories. The path-list specifies the directories and files that du reports on.

Options

Without any options, du displays the total storage used for each argument in path-list. For directories du displays this total after recursively listing the totals for all of the subdirectories.

a | (all) Displays the space used by all ordinary files along with the total for each directory.
c | Displays a grand total at the end of the output.
d | depth
 | Displays information for subdirectories to a level of depth directories.
h | (human readable) Displays sizes in K (kilobyte), M (megabyte), and G (gigabyte) blocks, as appropriate. Uses powers of 1,024.
H | (dereference) Includes the files that symbolic links given on the command line point to, not the symbolic links themselves. Does not include the files that symbolic links found while descending a directory hierarchy point to.
k | (kilobytes) Displays sizes in 1-kilobyte blocks.
L | (dereference) Includes the files that symbolic links point to, not the symbolic links themselves. Includes the files that symbolic links found while descending a directory hierarchy point to.
P | (no dereference) Includes the sizes of symbolic links, not the files that the links point to (default).
s | (summarize) Displays only the total size of each directory in path-list.
x | Reports only on files and directories in filesystems that hold the files and directories specified in path-list.

Notes

The du utility supports the BLOCKSIZE environment variable (page 663). The default is 512 bytes. The k option overrides BLOCKSIZE.

Examples

In the first example, du displays size information about subdirectories in the working directory. The last line contains the grand total for the working directory and its subdirectories.

$ du
26        ./Postscript
4         ./RCS
47        ./XIcon
4         ./Printer/RCS
12        ./Printer
105       .

The total (105) is the number of blocks occupied by all ordinary files and directories under the working directory. All files are counted, even though du displays only the sizes of directories.

Next, using the s option, du displays the total for each of the directories in /Users but not for any subdirectories:

# du -s  /Users/*
8       /Users/Shared
3064    /Users/alex
8832    /Users/zach

The c option causes du to display a grand total:

# du -s -c  /Users/*
8       /Users/Shared
...
8832    /Users/zach
20326   total

If you do not have read permission for a file or directory that du encounters, du sends a warning to standard error and skips that file or directory. The following example uses the s (summarize), h (human-readable), and c (total) options:

$ du -shc  /usr/*
107M     /usr/X11R6
156M     /usr/bin
 61M     /usr/include
du:  /usr/lib/jabber: Permission denied
 88M     /usr/lib
 76M     /usr/libexec
280K     /usr/local
du:  /usr/sbin/authserver: Permission denied
 39M     /usr/sbin
283M     /usr/share
344K     /usr/standalone
812M     total

The final example displays, in human-readable format, the total size of all files the user can read in the /usr filesystem. Redirecting standard error to /dev/null (page 444) discards all warnings about files and directories that are unreadable.

$ du -h -s /usr 2>/dev/null
812M     /usr

echo: Displays a message

echo [options] message

The echo utility copies its arguments, followed by a NEWLINE, to standard output. The Bourne Again Shell and TC Shell each has an echo builtin that works similarly to the echo utility.

Arguments

The message consists of one or more arguments, which can include quoted strings, ambiguous file references, and shell variables. A SPACE separates each argument from the next. The shell recognizes unquoted special characters in the arguments. For example, the shell expands an asterisk into a list of filenames in the working directory.

Options

You can configure the tcsh echo builtin to treat backslash escape sequences and the n option in different ways. Refer to echo_style in the tcsh man page. The typical tcsh configuration recognizes the n option.

n | (newline) Suppresses the NEWLINE terminating the message.
e | (escape) Uses escape notation. Available only with the bash echo builtin.

Notes

You can use echo to send messages to the screen from a shell script. See page 134 for a discussion of how to use echo to display filenames using wildcard characters.

The echo builtins in tcsh and bash provide an escape notation to represent certain nonprinting characters in message (Table VI-13). You must use the e option for backslash escape sequences to work with the bash echo builtin. Typically you do not need the e option with the tcsh echo builtin. The echo utility recognizes only \c.

Table VI-13. Backslash escape sequences

Sequence

Meaning

\a

Bell

\c

Suppress trailing NEWLINE

\n

NEWLINE

\t

HORIZONTAL TAB

\v

VERTICAL TAB

\\

BACKSLASH

Examples

The following examples contain messages with the backslash escape sequence \c. In the first example, the shell processes the arguments before calling echo. When the shell sees the \c, it replaces the \c with the character c. The next three examples show how to quote the \c so that the shell passes it to echo, which then does not append a NEWLINE to the end of the message. These examples run under bash and require the e option.

$ echo -e There is a newline after this.\c
There is a newline after this.c

$ echo -e 'There is no newline after this.\c'
There is no newline after this.$

$ echo -e "There is no newline after this.\c"
There is no newline after this.$

$ echo -e There is no newline after this.\\c
There is no newline after this.$

Under bash you can use the n option in place of e and the trailing \c.

The final example runs under tcsh, which typically uses n to suppress a trailing NEWLINE:

$ tcsh
tcsh $ echo -n There is no newline after this.
There is no newline after this.$

expr: Evaluates an expression

expr expression

The expr utility evaluates an expression and sends the result to standard output. It evaluates character strings that represent either numeric or nonnumeric values. Operators are used with the strings to form expressions.

Arguments

The expression is composed of strings interspersed with operators. Each string and operator constitute a distinct argument that you must separate from other arguments with a SPACE. You must quote operators that have special meanings to the shell (such as the multiplication operator, *).

The following list of expr operators is given in order of decreasing precedence. Each operator within a group of operators has the same precedence. You can change the order of evaluation by using parentheses.

: | (comparison) Compares two strings, starting with the first character in each string and ending with the last character in the second string. The second string is a regular expression with an implied caret (^) as its first character. If there is a match, expr displays the number of characters in the second string. If there is no match, expr displays a zero.
* | (multiplication)
/ | (division)
% | (remainder)
 | Work only on strings that contain the numerals 0 through 9 and optionally a leading minus sign. Convert strings to integer numbers, perform the specified arithmetic operation on numbers, and convert the result back to a string before sending it to standard output.
+ | (addition)
- | (subtraction)
 | Function in the same manner as the preceding group of operators.
< | (less than)
<= | (less than or equal to)
= or == | (equal to)
!= | (not equal to)
>= | (greater than or equal to)
> | (greater than)
 | Relational operators work on both numeric and nonnumeric arguments. If one or both of the arguments are nonnumeric, the comparison is nonnumeric and relies on the machine collating sequence (typically ASCII). If both arguments are numeric, the comparison is numeric. The expr utility displays 1 (one) if the comparison is true and 0 (zero) if it is false.
& | (AND) Evaluates both of its arguments. If neither is zero or a null string, expr displays the value of the first argument. Otherwise, it displays 0 (zero). You must quote this operator.
| | (OR) Evaluates the first argument. If it is neither zero nor a null string, expr displays the value of the first argument. Otherwise, it displays the value of the second argument. You must quote this operator.

Notes

The expr utility returns an exit status of 0 (zero) if the expression evaluates to other than a null string or the number 0, a status of 1 if the expression is null or 0, and a status of 2 if the expression is invalid.

Although expr and this discussion distinguish between numeric and nonnumeric arguments, all arguments to expr are nonnumeric (character strings). When applicable, expr attempts to convert an argument to a number (for example, when using the + operator). If a string contains characters other than 0 through 9 with an optional leading minus sign, expr cannot convert it. Specifically, if a string contains a plus sign or a decimal point, expr considers it to be nonnumeric. If both arguments are numeric, the comparison is numeric. If one is nonnumeric, the comparison is lexicographic.

Examples

The following examples show command lines that call expr to evaluate constants. You can also use expr to evaluate variables in a shell script. The fourth command displays an error message because of the illegal decimal point in 5.3.

$ expr 17 + 40
57
$ expr 10 - 24
-14
$ expr -17 + 20
3
$ expr 5.3 + 4
expr: non-numeric argument

The multiplication (*), division (/), and remainder (%) operators provide additional arithmetic power. You must quote the multiplication operator (precede it with a backslash) so that the shell will not treat it as a special character (an ambiguous file reference). You cannot put quotation marks around the entire expression because each string and operator must be a separate argument.

$ expr 5 \* 4
20
$ expr 21 / 7
3
$ expr 23 % 7
2

The next two examples show how parentheses change the order of evaluation. You must quote each parenthesis and surround the backslash/parenthesis combination with SPACEs:

$ expr 2 \* 3 + 4
10
$ expr 2 \* \( 3 + 4 \)
14

You can use relational operators to determine the relationship between numeric or nonnumeric arguments. The following commands compare two strings to see if they are equal; expr displays 0 when the relationship is false and 1 when it is true:

$ expr fred == sam
0
$ expr sam == sam
1

In the following examples, the relational operators, which must be quoted, establish an order between numeric or nonnumeric arguments. If a relationship is true, expr displays a 1.

$ expr fred \> sam
0
$ expr fred \< sam
1
$ expr 5 \< 7
1

The next command compares 5 with m. When one of the arguments that expr is comparing with a relational operator is nonnumeric, expr considers the other to be nonnumeric. In this case, because m is nonnumeric, expr TReats 5 as a nonnumeric argument. The comparison is between the ASCII (on many systems) values of m and 5. The ASCII value of m is 109 and that of 5 is 53, so expr evaluates the relationship as true.

$ expr 5 \< m
1

The next example shows the matching operator determining that the four characters in the second string match the first four characters in the first string. The expr utility displays the number of matching characters (4).

$ expr abcdefghijkl : abcd
4

The & operator displays 0 if one or both of its arguments are 0 or a null string; otherwise, it displays the first argument:

$ expr '' \& book
0

$ expr magazine \& book
magazine
$ expr 5 \& 0
0

$ expr 5 \& 6
5

The | operator displays the first argument if it is not 0 or a null string; otherwise, it displays the second argument:

$ expr '' \| book
book

$ expr magazine \| book
magazine

$ expr 5 \| 0
5

$ expr 0 \| 5
5

$ expr 5 \| 6
5

file: Displays the classification of a file

file [option] file-list

The file utility classifies files according to their contents.

Arguments

The file-list specifies the pathnames of one or more files that file classifies. You can specify any kind of file, including ordinary, directory, and special files, in the file-list.

Options

files-from file | f | file
 |  | Takes the names of files to be examined from file rather than from file-list on the command line. The names of the files must be listed one per line in file.
mime | i | Displays MIME (page 943) type strings.
help |  | Displays a help message.
dereference | L | Reports on the files that symbolic links point to, not on the symbolic links themselves. The default is to report on symbolic links (not to dereference files).
uncompress | z | (zip) Attempts to classify files found within a compressed file.

Notes

The file utility can classify more than 5,000 file types. Some of the more common file types found on Mac OS X systems, as displayed by file, are listed here:

archive
ascii text
c program text
commands text
core file
cpio archive
data
directory
Mach-O executable ppc
empty
English text
executable

The file utility uses a maximum of three tests in its attempt to classify a file: filesystem, magic number, and language tests. When file identifies the type of a file, it ceases testing. The filesystem test examines the return from a stat system call to see whether the file is empty or a special file. The magic number (page 941) test looks for data in particular fixed formats near the beginning of the file. The language test, if needed, determines whether the file is a text file, what kind of encoding it uses, and what language it is written in. Refer to the file man page for a more detailed description of how file works. The results of file are not always correct.

The file utility does not examine Mac file type and creator codes (page 95).

Examples

Some examples of file identification follow:

tabs:             Mach-O executable ppc
tabs.c:           ASCII C program text
bin:              directory
dscn0436.jpg:     JPEG image data, EXIF standard 2.1
0-6.cwk:          data
crop0151.tiff:    TIFF image data, big-endian
dohlynx.png:      PNG image data, 639 x 431, 8-bit grayscale, non-interlaced

find: Finds files based on criteria

find [options] [directory-list] [expression]

The find utility selects files that are located in specified directory hierarchies and that meet specified criteria.

Arguments

The directory-list specifies which directories find will search. When find searches a directory, it searches all subdirectories to all levels.

The expression contains criteria, as described in the "Criteria" section. The find utility tests each of the files in each of the directories in the directory-list to see whether it meets the criteria described by the expression. When you do not specify an expression, the expression defaults to print.

A SPACE separating two criteria is a Boolean AND operator: The file must meet both criteria to be selected. A or or o separating the criteria is a Boolean OR operator: The file must meet one or the other (or both) of the criteria to be selected.

You can negate any criterion by preceding it with an exclamation point. The find utility evaluates criteria from left to right unless you group them using parentheses.

Within the expression you must quote special characters so that the shell does not interpret them but rather passes them to find. Special characters that you may frequently use with find include exclamation points, parentheses, brackets, question marks, and asterisks.

Each element within the expression is a separate argument. You must separate arguments from each other with SPACEs. A SPACE must appear on both sides of each parenthesis, exclamation point, criterion, or other element.

Options

d | (depth) Takes action on entries in a directory before it acts on the directory itself. When you use find to send filenames to cpio or pax, this option enables cpio or pax to preserve the modification times of directories when you restore files (assuming that you use the m option to cpio; pax preserves modification times by default). See the "Discussion" (page 695) and "Examples" (page 696) sections under cpio.
H | (dereference) Selects the files that symbolic links given on the command line point to, not the symbolic links themselves. Does not select the files that symbolic links found while descending a directory hierarchy point to.
L | (dereference) Selects the files that symbolic links point to, not the symbolic links themselves. Includes the files that symbolic links found while descending a directory hierarchy point to.
s | (sort) Traverses directories in lexicographical (dictionary) order.
x | (xdev) Prevents find from searching directories in filesystems other than the filesystem(s) that hold the directories it was explicitly told to search.
x | Searches only directories in filesystems that hold the directories specified in directory-list.

Criteria

You can use the following criteria within the expression. In this list, ±n is a decimal integer that can be expressed as +n (more than n), n (fewer than n), or n (exactly n).

anewer filename

(accessed newer) The file being evaluated meets this criterion if it was accessed more recently than filename.

atime ±n

(access time) The file being evaluated meets this criterion if it was last accessed ±n days ago. When you use this option, find changes the access times of directories it searches.

exec command \;

The file being evaluated meets this action criterion if the command returns an exit status of 0 (zero [true]). You must terminate the command with a quoted semicolon. A pair of braces ({}) within the command represents the name of the file being evaluated. You can use the exec action criterion at the end of a group of other criteria to execute the command if the preceding criteria are met. Refer to "Discussion" for more information. See the "Examples" under xargs on page 893 for a more efficient way of doing what this option does.

group name

The file being evaluated meets this criterion if it is associated with the group named name. You can use a numeric group ID in place of name.

inum n

The file being evaluated meets this criterion if its inode number is n.

links ±n

The file being evaluated meets this criterion if it has ±n links.

mtime ±n

(modify time) The file being evaluated meets this criterion if it was last modified ±n days ago.

name filename

The file being evaluated meets this criterion if the pattern filename matches its name. The filename can include wildcard characters (*, ?, and []) but these characters must be quoted.

newer filename

The file being evaluated meets this criterion if it was modified more recently than filename.

nogroup

The file being evaluated meets this criterion if it does not belong to a group in the system group database.

nouser

The file being evaluated meets this criterion if it does not belong to a user in the passwd database.

ok command \;

This action criterion is the same as exec with two exceptions: It displays each command to be executed enclosed in angle brackets, and it executes the command only if it receives a response that starts with a y or Y from standard input.

perm [±]mode

The file being evaluated meets this criterion if it has the access permissions given by mode. If mode is preceded by a minus sign (-), the file access permissions must include all the bits in mode. If mode is preceded by a plus sign (+), the file access permissions must include at least one of the bits in mode. If no plus or minus sign precedes mode, the mode of the file must exactly match mode. You may use either a symbolic or an octal representation for mode (see chmod on page 676).

print

The file being evaluated always meets this action criterion. When evaluation of the expression reaches this criterion, find displays the pathname of the file it is evaluating. If print is the only criterion in the expression, find displays the names of all files in the directory-list. If this criterion appears with other criteria, find displays the name only if the preceding criteria are met. If no action criteria appear in the expression, print is assumed. (Refer to the following "Discussion" and "Notes" sections.)

print0

Similar to print, but follows the pathname with a null character instead of a NEWLINE. Useful with xargs (see the example on page 894).

size ±n[c|k]

The file being evaluated meets this criterion if it is the size specified by ±n, as measured in 512-byte blocks. Follow n with the letter c to measure the size of files in characters.

type filetype

The file being evaluated meets this criterion if its file type is specified by filetype. Select a filetype from the following list:

b Block special file

c Character special file

d Directory file

f Ordinary file

l Symbolic link

p FIFO (named pipe)

s Socket

user name

The file being evaluated meets this criterion if it belongs to the user with the username name. You can use a numeric user ID in place of name.

Discussion

Assume that x and y are criteria. The following command line never tests whether the file meets criterion y if it does not meet criterion x. Because the criteria are separated by a SPACE (the Boolean AND operator), once find determines that criterion x is not met, the file cannot meet the criteria so find does not continue testing. You can read the expression as "(test whether) the file meets criterion x and (SPACE means and) criterion y."

$ find dir x y

The next command line tests the file against criterion y if criterion x is not met. The file can still meet the criteria so find continues the evaluation. You can read the expression as "(test whether) the file meets criterion x or criterion y." If the file meets criterion x, find does not evaluate criterion y as there is no need to do so.

$ find dir x -or y

Action criteria

Certain "criteria" do not select files but rather cause find to take action. The action is triggered when find evaluates one of these action criteria. Therefore the position of an action criterion on the command linenot the result of its evaluationdetermines whether find takes the action.

The print action criterion causes find to display the pathname of the file it is testing. The following command line displays the names of all files in the dir directory (and all its subdirectories), regardless of whether they meet the criterion x:

$ find dir -print x

The following command line displays only the names of the files in the dir directory that meet criterion x:

$ find dir x -print

This common use of print after the testing criteria is the default action criterion. The following command line generates the same output as the previous one:

$ find dir x

Notes

You can use the a operator between criteria to improve readability. This operator is a Boolean AND operator, just as the SPACE is.

In earlier versions of find, the d, H, and x options were implemented as action criteria named depth, follow, and xdev, respectively.

Examples

The simplest find command has only a directory as an argument and lists the files in the specified directory and all subdirectories. The first example lists the contents of the working directory hierarchy; it uses a period to designate the working directory:

$ find .
...

The following command finds the files in the working directory and subdirectories that have filenames beginning with a. To prevent the shell from interpreting a* as an ambiguous file reference, it is enclosed within quotation marks.

$ find . -name 'a*'

The print criterion is implicit in the preceding commands.

The next command sends a list of selected filenames to the pax utility, which writes them to an archive file. (Under Mac OS X 10.3 and earlier, the pax utility did not preserve extended attributes, so this approach is not a suitable way to back up all types of files.) The first part of the command line ends with a pipe symbol, so the shell expects another command to follow and displays a secondary prompt (>) before accepting the rest of the command line. You can read this find command as "find, in the root directory and all subdirectories (/), ordinary files (type f) that have been modified within the past day (mtime 1), with the exception of files whose names are suffixed with .o (! name '* .o')." (An object file carries a .o suffix and usually does not need to be preserved because it can be re-created from the corresponding source file.)

$ find / -type f -mtime -1 ! -name '*.o' -print |
> pax -w > /Volumes/Backups/recent.pax

The following command finds, displays the filenames of, and deletes the files in the working directory and subdirectories named *.bak or junk:

$ find . \( -name "*.bak" -o -name junk \) -print -exec rm {} \;
...

The parentheses and the semicolon following exec are quoted so that the shell does not treat them as special characters. SPACEs separate the quoted parentheses from other elements on the command line. Read this find command as "find, in the working directory and subdirectories (.), files named *.bak (name "*.bak") or (o) junk (name junk) [if a file meets these criteria, continue] and (SPACE) print the name of the file (print) and (SPACE) delete the file (exec rm {})."

The next shell script uses find in conjunction with grep to identify files that contain a particular string. This script enables you to look for a file when you remember its contents but cannot recall its filename. The findit script locates files in the working directory and subdirectories that contain the string specified on the command line. The type f criterion is necessary so that find will pass to grep only the names of ordinary files, not directory files.

$ cat findit
find . -type f -exec grep -l "$1" {} \;
$ findit "Executive Meeting"
 ./january/memo.0102
 ./april/memo.0415

When called with the string Executive Meeting, findit locates two files containing that string: ./january/memo.0102 and ./april/memo.0415. The period (.) in the pathnames represents the working directory; january and april are subdirectories of the working directory. The grep utility with the recursive option performs the same function as the findit script.

The next command looks in two directories for files that are larger than 100 blocks (size +100) and have been accessed more than five days agothat is, the files have not been accessed within the past five days (atime +5). This find command then asks whether you want to delete the file (ok rm {}). You must respond to each query with y (yes) or n (no). The rm command works only if you have write and execute access permission to the directory.

$ find /Users/alex /Users/barbara -size +100 -atime +5 -ok rm {} \;
< rm ... /Users/alex/notes >? y
< rm ... /Users/alex/letter >? n
...

In the next example, /Users/alex/memos is a symbolic link to Jenny's directory named /Users/jenny/memos. When you use the H option, find follows the symbolic link and searches the contents of that directory.

$ ls -l /Users/alex
lrwxr-xr-x  1 alex  alex  17 Aug 19 17:07 memos -> /Users/jenny/memos
-rw-r--r--  1 alex  alex 5119 Aug 19 17:08 report

$ find /Users/alex -print
/Users/alex
/Users/alex/memos
/Users/alex/report
/Users/alex/.profile

$ find -H /Users/alex -print
/Users/alex
/Users/alex/memos
/Users/alex/memos/memo.817
/Users/alex/memos/memo.710
/Users/alex/report
/Users/alex/.profile

finger: Displays information about users

finger [options] [user-list]

The finger utility displays the usernames of users, together with their full names, terminal device numbers, times they logged in, and other information. It can retrieve information from both local and remote systems.

Arguments

Without any arguments, finger provides a short (s) report on users who are logged in on the local system. When you specify a user-list, which is a list of usernames that is not case sensitive, finger provides a long (l) report on each user in the user-list.

If a name starts with an at sign (@), finger interprets the name following the @ as the name of a remote host and reports on the users on that system. If a username precedes the @ sign, finger provides information on that user on the remote system.

Options

The options control how much information finger displays.

l | (long) Displays detailed information (the default when you specify a user-list).
m | (match) If a user-list is specified, displays entries only for those users whose username matches one of the names in user-list. Without this option, the user-list names match usernames and full names.
p | (no plan or project) Does not display the contents of .plan or .project files for users. Because these files may contain backslash escape sequences that can change the behavior of the screen, you may not wish to view them. Normally the long listing displays the contents of these files if they exist in the user's home directory.
s | (short) Provides a short report for each user (the default when you do not specify a user-list).

Discussion

The long report includes the user's username, full name, home directory location, and login shell, plus information about when the user last logged in and how long it has been since the user last typed on the keyboard and read her email. After displaying this information, finger displays the contents of the ~/.plan and ~/.project files in the user's home directory. It is up to each user to create and maintain these files, which usually provide more information about the user (such as telephone number, postal mail address, schedule, and interests).

The short report generated by finger is similar to that provided by the w utility. It includes the user's username, his full name, the device number of the user's terminal, the amount of time that has elapsed since the user last typed on the terminal keyboard, the time the user logged in, and, if the user logged in over the network, the name of the remote system.

Notes

When you specify a network address, the finger utility queries a standard network service that runs on the remote system. Although this service is supplied with Mac OS X systems, it is disabled by default. Some administrators choose not to enable it so as to minimize the load on their systems, to eliminate possible security risks, or simply to maintain privacy. If you try to use finger to get information on someone at such a site, the result may be an error message or nothing at all. The remote system determines how much information to share with the local system and in what format. As a result the report displayed for any given system may differ from the examples shown here. See also "finger: Lists Users on the System" on page 61.

.nofinger

A file named ~/.nofinger causes finger to deny the existence of the person whose home directory it appears in. For this subterfuge to work, the fingerd daemon or the finger utility must be able to see the .nofinger file (generally the home directory must have its execute bit for other users set).

Examples

The first example displays information on the users logged in on the local system:

$ finger
Login    Name                    TTY   Idle  Login Time   Office     Phone
alex     Alex Watson            *con    1d    Jun 25 21:03
hls      Helen Simpson          *p1    13:29  Jun 25 21:02)
jenny    Jenny Chen              p2           Jun 26 07:47

The asterisks (*) in front of the names of Helen and Alex's terminal (TTY) lines indicate that they have blocked others from sending messages directly to their screens (see mesg on page 64). A long report displays the string messages off for users who have disabled messages.

The next two examples cause finger to contact the remote system named kudos over the network for information:

$ finger @kudos
[kudos]
Login    Name           TTY    Idle   Login  Time     Office     Phone
alex     Alex Watson    tty1   23:15  Jun 25 11:22
roy      Roy Wong       pts/2         Jun 25 11:22

$ finger watson@kudos
[kudos]
Login: alex                             Name: Alex Watson
Directory: /Users/alex                   Shell: /bin/zsh
On since Sat Jun 25 11:22 (PDT) on tty1,  idle 23:22
Last login Sun Jun 26 06:20 (PDT) on ttyp2 from speedy
Mail last read Thu Jun 23 08:10 2005 (PDT)
Plan:
For appointments contact Jenny Chen, x1963.

fmt: Formats text very simply

fmt [option] [file-list]

The fmt utility does simple text formatting, attempting to make all nonblank lines nearly the same length.

Arguments

The fmt utility reads the files in file-list and sends a formatted version of their contents to standard output. If you do not specify a filename or if you specify a filename of , fmt reads from standard input.

Options

c | (center) Centers each line. This option suppresses line splitting and joining.
n | (nroff) Formats lines beginning with a period (.). The default is to leave these lines alone, for compatibility with the nroff text formatter.
s | (spaces) Replaces multiple whitespace characters with a single SPACE.
w | n
 | (width) Changes the output line length to n characters. Without this option, fmt keeps output lines close to 75 characters long. You can also specify this option as n.

Notes

The fmt utility works by moving NEWLINE characters. The indention of lines, as well as the spacing between words, is left intact.

This utility is often used to format text while you are using an editor, such as vim. For example, you can format a paragraph with the vim editor in Command mode by positioning the cursor at the beginning of the paragraph and giving the command !}fmt60. This command replaces the paragraph with the output generated by feeding it through fmt, specifying a line length of 60 characters. Type u immediately if you want to undo the formatting.

Examples

The following example shows how fmt attempts to make all lines the same length. The w 50 option gives a target line length of 50 characters.

$ cat memo
One factor that is important to remember while administering the dietary
intake of Charcharodon carcharias is that there is, at least from
the point of view of the subject,
very little
differentiating the prepared morsels being proffered from your digits.

In other words, don't feed the sharks!
$ fmt -w 50 memo
One factor that is important to remember while
administering the dietary intake of Charcharodon
carcharias is that there is, at least from the
point of view of the subject, very little
differentiating the prepared morsels being
proffered from your digits.

In other words, don't feed the sharks!

ftp: transfers files over a network

ftp [options] [remote-system]

The ftp utility is a user interface to the standard File Transfer Protocol (FTP), which transfers files between systems that can communicate over a network. To establish an FTP connection, you must have an account (personal, guest, or anonymous) on the remote system.

Security: Use FTP only to download public information

FTP is not a secure protocol. The ftp utility sends your password over the network as cleartext, which is not a secure practice. You can use scp (page 832) for many FTP functions other than allowing anonymous users to download information. Because scp uses an encrypted connection, user passwords and data cannot be sniffed easily.

Arguments

The remote-system is typically the name or network address of the server that is running an ftpd daemon and that you want to exchange files with. In this case ftp connects to the server. If remote-system is a URL (including the protocol such as http:// or ftp://), ftp downloads the file(s) specified by the URL. Without any arguments, ftp runs interactively.

Options

i | (interactive) Turns off prompts during file transfers with mget and mput. See also prompt.
n | (no automatic login) Disables automatic logins.
v | (verbose) Tells you more about how ftp is working. Displays responses from the remote-system and reports transfer times and speeds.

Discussion

When called with an argument that is a URL, ftp attempts to fetch the named file automatically. If the URL uses the FTP protocol and specifies a directory, ftp connects to the server, cds to that directory, and displays a prompt.

When called with an argument that is the name or network address of a server, or when called without an argument, ftp is interactive. After you start it, ftp prompts you to enter commands to set parameters and transfer files. You can use a number of commands in response to the ftp> prompt; following are some of the more common ones.

![command]

Escapes to (spawns) a shell on the local system (use CONTROL-D or exit to return to ftp when you are finished using the local shell). Follow the exclamation point with a command to execute that command only; ftp returns you to the ftp> prompt when the command completes executing. Because the shell that ftp spawns with this command is a child of the shell that is running ftp, changes you make in this shell are not preserved when you return to ftp. Specifically, when you want to copy files to a local directory other than the directory that you started ftp from, you need to use the ftp lcd command to change the local working directory: Issuing a cd command in the spawned shell will not make the desired change. See "lcd (local cd)" on page 743 for an example.

ascii | Sets the file transfer type to ASCII. Allows you to transfer text files from systems that end lines with a RETURN/LINEFEED combination and automatically strip off the RETURN. This type of transfer is useful when the remote computer is a DOS or Windows machine. The cr command must be on for ascii to work.
binary | Sets the file transfer type to binary (default). Allows you to transfer files that contain non-ASCII (unprintable) characters correctly. This option also works for ASCII files that do not require changes to the ends of lines.
bye | Closes the connection to a remote system and terminates ftp. Same as quit.
cd remote-directory
 | Changes to the working directory named remote-directory on the remote system.
close | Closes the connection with the remote system without exiting from ftp.
cr | Toggles RETURN stripping when you retrieve files in ASCII mode. See ascii.
dir [directory [file]]
 | Displays a listing of the directory named directory from the remote system. When you do not specify directory, the working directory is displayed. When you specify file, the listing is saved on the local system in a file named file.
get remote-file [local-file]
 | Copies remote-file to the local system under the name local-file. Without local-file, ftp uses remote-file as the filename on the local system. The remote-file and local-file names can be pathnames.
glob | Toggles filename expansion for mget and mput commands and displays the current state (Globbing on or Globbing off).
help | Displays a list of commands recognized by the ftp utility on the local system.
lcd [local_directory]
 | (local change directory) Changes the working directory on the local system to local_directory. Without an argument, changes the working directory on the local system to your home directory (just as cd does without an argument).
ls [directory [file]] | Similar to dir but produces a more concise listing on some remote systems. When you specify file, the listing is saved on the local system in a file named file.
mget remote-file-list
 | (multiple get) Unlike the get command, the mget command can retrieve multiple files from the remote system. You can name the remote files literally or use wildcards (see glob). See also prompt.
mput local-file-list
 | (multiple put) The mput command can copy multiple files from the local system to the remote system. You can name the local files literally or use wildcards (see glob). See also prompt.
open | Interactively specifies the name of the remote system. This command is useful if you did not specify a remote system on the command line or if the attempt to connect to the system failed.
passive | Toggles between active (PORTthe default) and passive (PASV) transfer modes and displays the transfer mode. See "Passive versus active connections" under "Notes."
prompt | When using mget or mput to receive or send multiple files, ftp asks for verification (by default) before transferring each file. This command toggles that behavior and displays the current state (Interactive mode off or Interactive mode on).
put local-file [remote-file]
 | Copies local-file to the remote system under the name remote-file. Without remote-file, ftp uses local-file as the filename on the remote system. The remote-file and local-file names can be pathnames.
pwd | Causes ftp to display the pathname of the remote working directory. Use !pwd to display the name of the local working directory.
reget remote-file | Attempts to resume an aborted transfer. Similar to get, but instead of overwriting an existing local file, ftp appends new data to it. Not all servers support reget.
quit | Quits the ftp session. Same as bye.
user [username] | If the ftp utility did not log you in automatically, you can specify your account name as username. Without username, ftp prompts you for your username.

Notes

A system running ftp can exchange files with any of the many operating systems that support the FTP protocol. Numerous sites offer archives of free information on an FTP server, although many of them are merely alternatives to easier-to-access Web sites (for example, ftp://ftp.ibiblio.org/pub and http://www.ibiblio.org/pub). Most browsers can connect to and download files from FTP servers.

The ftp utility makes no assumptions about filesystem naming or structure because you can use ftp to exchange files with non-UNIX systems (whose filename conventions may be different).

Anonymous FTP

Many systemsmost notably those from which you can download free softwareallow you to log in as anonymous. Most systems that support anonymous logins accept the name ftp as an easier-to-spell and quicker-to-enter synonym for anonymous. An anonymous user is usually restricted to a portion of a filesystem set aside to hold files that are to be shared with remote users. When you log in as an anonymous user, the server prompts you to enter a password. Although any password may be accepted, by convention you are expected to supply your email address. Many systems that permit anonymous access store interesting files in the pub directory.

Passive versus active connections

A client can ask an FTP server to establish either a PASV (passivethe default) or a PORT (active) connection for data transfer. Some servers are limited to one type of connection. The difference between a passive and an active FTP connection lies in whether the client or the server initiates the data connection. In passive mode, the client initiates the data connection to the server (on port 20 by default); in active mode, the server initiates the data connection (there is no default port). Neither type of connection is inherently more secure than the other. Passive connections are more common because a client behind a NAT (page 944) firewall can connect to a passive server and because it is simpler to program a scalable passive server.

Automatic login

You can store server-specific FTP username and password information so that you do not have to enter it each time you visit an FTP site. Each line of the ~/.netrc file identifies a server. When you connect to an FTP server, ftp reads ~/.netrc to determine whether you have an automatic login set up for that server. The format of a line in .netrc is

machine server login username password passwd

where server is the name of the server, username is your username, and passwd is your password on server. Replace machine with default on the last line of the file to specify a username and password for systems not listed in ~/.netrc. The default line is useful for logging in on anonymous servers. A sample ~/.netrc file follows:

$ cat ~/.netrc
machine bravo login alex password mypassword
default login anonymous password alex@example.com

Protect the account information in .netrc by making it readable by only the user whose home directory it appears in. Refer to the netrc man page for more information.

Examples

Following are two ftp sessions wherein Alex transfers files from and to an FTP server named bravo. When Alex gives the command ftp bravo, the local ftp client connects to the server, which asks for a username and password. Because he is logged in on his local system as alex, ftp suggests that he log in on bravo as alex. To log in as alex, Alex could just press RETURN. His username on bravo is watson, however, so he types watson in response to the Name (bravo:alex): prompt. Alex responds to the Password: prompt with his normal system password, and the FTP server greets him and informs him that it is Using binary mode to transfer files. With ftp in binary mode, Alex can transfer ASCII and binary files.

Connect and log in

$ ftp bravo
Connected to bravo.
220 bravo.example.com FTP server (NetBSD-ftpd 20040809) ready.
Name (bravo:alex): watson
331 Password required for watson.
Password:
230-
    NetBSD 2.0.2 (BRAVO) #0: Sun Feb 20 06:11:02 CST 2005

    Welcome to NetBSD!
230 User watson logged in.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>

After logging in, Alex uses the ftp ls command to display the contents of his remote working directory, which is his home directory on bravo. Then he cds to the memos directory and displays the files there.

ls and cd

ftp> ls
229 Entering Extended Passive Mode (|||63979|)
150 Opening ASCII mode data connection for '/bin/ls'.
total 6
drwxr-xr-x    2 500      500          512 Oct 10 23:52 expenses
drwxr-xr-x    2 500      500          512 Oct 10 23:59 memos
drwxrwxr-x   22 500      500          512 Oct 10 23:32 tech
226 Transfer complete.

ftp> cd memos
250 Directory successfully changed.

ftp> ls
229 Entering Extended Passive Mode (|||63979|)
150 Opening ASCII mode data connection for '/bin/ls'.
total 56
-rw-r--r--    1 500      500          4770 Oct 10 23:58 memo.0514
-rw-r--r--    1 500      500          7134 Oct 10 23:58 memo.0628
-rw-r--r--    1 500      500          9453 Oct 10 23:58 memo.0905
-rw-r--r--    1 500      500          3466 Oct 10 23:59 memo.0921
-rw-r--r--    1 500      500          1945 Oct 10 23:59 memo.1102
226 Transfer complete.

Next Alex uses the ftp get command to copy memo.1102 from the server to the local system. Binary mode ensures that he will get a good copy of the file regardless of whether it is binary or ASCII. The server confirms that the file was copied successfully and notes the size of the file and the time it took to copy. Alex then copies the local file memo.1114 to the remote system. The file is copied into his remote working directory, memos.

get and put

ftp> get memo.1102
local: memo.1102 remote: memo.1102
229 Entering Extended Passive Mode (|||64083|)
150 Opening BINARY mode data connection for 'memo.1102' (77 bytes).
100% |*************************************|    77    335.69 KB/s 00:00

226 Transfer complete.
77 bytes received in 00:00 (0.41 KB/s)
ftp> put memo.1114
local: memo.1114 remote: memo.1114
229 Entering Extended Passive Mode (|||64082|)
150 Opening BINARY mode data connection for 'motd'.
100% |*************************************|   974   458.50 KB/s  00:00
226 Transfer complete.
974 bytes sent in 00:00 (1.73 KB/s)

After a while Alex decides he wants to copy all the files in the memo directory on bravo to a new directory on his local system. He gives an ls command to make sure he is going to copy the right files, but ftp has timed out. Instead of exiting from ftp and giving another ftp command from the shell, Alex gives an open bravo command to reconnect to the server. After logging in, he uses the ftp cd command to change directories to memos on the server.

Timeout and open

ftp> ls
421 Timeout (900 seconds): closing control connection.
ftp> open bravo
Connected to bravo
...
ftp> cd memos
250 Directory successfully changed.

Icd (local cd)

At this point, Alex realizes he has not created the new directory to hold the files he wants to download. Giving an ftp mkdir command would create a new directory on the server, but Alex wants a new directory on the local system. He uses an exclamation point (!) followed by a mkdir memos.hold command to invoke a shell and run mkdir on the local system, creating a directory named memos.hold in his working directory on the local system. (You can display the name of the working directory on the local system with the command !pwd.) Next because he wants to copy files from the server to the memos.hold directory on his local system, Alex has to change his working directory on the local system. Giving the command !cd memos.hold will not accomplish what Alex wants to do because the exclamation point spawns a new shell on the local system and the cd command would be effective only in the new shell, which is not the shell that ftp is running under. For this situation, ftp provides the lcd (local cd) command, which changes the working directory for ftp and reports on the new local working directory.

ftp> !mkdir memos.hold

ftp> lcd memos.hold
Local directory now /Users/alex/memos.hold

Alex uses the ftp mget (multiple get) command followed by the asterisk (*) wildcard to copy all the files in the remote memos directory to the memos.hold directory on the local system. When ftp prompts him for the first file, he realizes that he forgot to turn off prompts, so he responds with n and presses CONTROL-C to stop copying files in response to the second prompt. The server checks whether he wants to continue with his mget command.

Next Alex gives the ftp prompt command, which toggles the prompt action (turns it off if it is on and turns it on if it is off). Now when he gives an mget * command, ftp copies all the files without prompting him.

After getting the files he wants, Alex gives a quit command to close the connection with the server, exit from ftp, and return to the local shell prompt.

mget and prompt

ftp> mget *
mget memo.0514 [anpqy]? n
mget memo.0628 [anpqy]? CONTROL-C
Continue with mget [anpqy]? n

ftp> prompt
Interactive mode off.
ftp> mget *
local: memo.0514 remote: memo.0514
229 Entering Extended Passive Mode (|||63989|)
150 Opening BINARY mode data connection for memo.0514 (4770 bytes).
100% |*************************************|   4770   97.65 KB/s   00:00
226 Transfer complete.
19 bytes received in 00:00 (0.09 KB/s)
local: memo.0628 remote: memo.0628
227 Entering Passive Mode (192,168,0,6,65,102)
150 Opening BINARY mode data connection for memo.0628 (7134 bytes).
100% |*************************************|  7134  147.62 KB/s   00:00
226 Transfer complete.
...
ftp> quit
221 Goodbye.

gcc: Compiles C, Objective-C, and C++ programs

gcc [options] file-list [larg]

g++ [options] file-list [larg]

Mac OS X uses the GNU C compiler, gcc, to preprocess, compile, assemble, and link C and Objective-C language source files. The same compiler with a different front end, g++, processes C++ source code. The gcc and g++ compilers can also assemble and link assembly language source files, link object files only, or build object files for use in shared libraries.

The input to these compilers comes from files you specify on the command line. Unless you use the o option, they store the executable program in a.out.

The gcc and g++ compilers are part of GCC, the GNU Compiler Collection, which includes front ends for C, C++, Objective-C, Fortran, Java, and Ada as well as libraries for these languages. Go to gcc.gnu.org for more information.

Tip: gcc and g++

Although this section specifies the gcc compiler, most of it applies to g++ as well.

Arguments

The file-list specifies the pathnames of one or more files that gcc is to process.

Options

Without any options, gcc accepts C language source files, assembly language files, object files, and other files described in Table VI-14 on page 747. The gcc utility preprocesses, compiles, assembles, and links these files as appropriate, producing an executable file named a.out. If gcc is used to create object files without linking them to produce an executable file, each object file is named by adding the extension .o to the basename of the corresponding source file. If gcc is used to create an executable file, it deletes the object files after linking.

Table VI-14. Filename extensions

Extension

Type of file

.a

Library of object modules

.c

C language source file

.C, .cc, or .cxx

C++ language source file

.i

Preprocessed C language source file

.ii

Preprocessed C++ language source file

.m

Objective-C language source file

.o

Object file

.s

Assembly language source file

.S

Assembly language source file that needs preprocessing

This section lists some of the most commonly used options. When certain filename extensions are associated with an option, you can assume that the extension is added to the basename of the source file.

c | (compile) Suppresses the linking step of compilation. Compiles and/or assembles source code files and leaves the object code in files with the extension .o.
Dname[=value]
 | Usually #define preprocessor directives are given in header, or include, files. You can use this option to define symbolic names on the command line instead. For example, DOSX is equivalent to having the line #define OSX in an include file, and DMACH=PPC is the same as #define MACH PPC.
E | (everything) For source code files, suppresses all steps of compilation except preprocessing and writes the result to standard output. By convention the extension .i is used for preprocessed C source code files and .ii for preprocessed C++ source code files.
fpic
 | Causes gcc to produce position-independent code, which is suitable for installation into a shared library.
fwritable-strings
 | By default the GNU C compiler places string constants into protected memory, where they cannot be changed. Some (usually older) programs assume that you can modify string constants. This option changes the behavior of gcc so that string constants can be modified.
g | (gdb) Embeds diagnostic information in the object files. This information is used by symbolic debuggers, such as gdb (page 501). Although it is necessary only if you later use a debugger, it is a good practice to include this option as a matter of course.
Idirectory
 | Looks for include files in directory before looking in the standard locations. Give this option multiple times to look in more than one directory.
larg
 | (lowercase "l") Searches the directory /usr/lib for a library file named libarg.a or libarg.dylib. If the file is found, gcc then searches this library for any required functions. Replace arg with the name of the library you want to search. For example, the lm option normally links the standard math library libm.dylib. The position of this option is significant: It generally needs to go at the end of the command line but can be repeated multiple times to search different libraries. Libraries are searched in the order in which they appear on the command line. The linker uses the library only to resolve undefined symbols from modules that precede the library option on the command line. You can add the paths of other library directories to search for libarg.a by using the L option.
Ldirectory
 | Adds directory to the list of directories to search for libraries given with the l option. Directories that are added to the list with L are searched before looking in the standard locations for libraries.
ofile
 | (output) Names the executable program that results from linking file instead of a.out.
On
 | (optimize) Attempts to improve (optimize) the object code produced by the compiler. The value of n may be 0, 1, 2, or 3; the default value is 1. Larger values of n result in better optimization but may increase both the size of the object file and the time it takes gcc to run. Using O0 turns off optimization. Many related options control precisely the types of optimizations attempted by gcc when you use O. Refer to the gcc info page for details.
pedantic
 | The C language accepted by the GNU C compiler includes features that are not part of the ANSI standard for the C language. Using this option forces gcc to reject these language extensions and accept only the standard C programming language features.
Q | Displays the names of functions as gcc compiles them. Also displays statistics about each pass.
S | (suppress) Suppresses the assembling and linking steps of compilation on source code files. The resulting assembly language files have .s filename extensions.
traditional
 | Causes gcc to accept only C programming language features that existed in the traditional Kernighan and Ritchie C programming language. This option allows you to compile correctly older programs written using the traditional C language that existed before the ANSI standard C language was defined.
Wall
 | Causes gcc to warn you about questionable code in the source code files. Many related options control warning messages more precisely. (See page 499.)

Notes

The preceding list of options represents only a fraction of the full set of options available with the GNU C compiler. See the gcc info page for a complete list.

See "Programming in C" on page 478 for more information about using the gcc compiler.

Although the o option is generally used to specify a filename in which to store object code, this option also allows you to name files resulting from other compilation steps. In the following example, the o option causes the assembly language produced by the gcc command to be stored in the file acode instead of pgm.s, the default:

$ gcc -S -o acode pgm.c

The lint utility found in many UNIX systems is not available under Mac OS X. However, the Wall option (page 499) performs many of the same checks and can be used in place of lint.

Table VI-14 summarizes the conventions used by the C compiler for assigning filename extensions.

Examples

The first example compiles, assembles, and links a single C program, compute.c. The executable output is put in a.out. The gcc utility deletes the object file.

$ gcc compute.c

The next example compiles the same program using the C optimizer (O option). It assembles and links the optimized code. The o option causes gcc to put the executable output in compute.

$ gcc -O -o compute compute.c

Next a C source file, an assembly language file, and an object file are compiled, assembled, and linked. The executable output goes in progo.

$ gcc -o progo procom.c profast.s proout.o

In the next example, gcc searches the standard math library stored in /lib/libm.a when it is linking the himath program and puts the executable output in a.out:

$ gcc himath.c -lm

In the following example, the C compiler compiles topo.c with options that check the code for questionable source code practices (Wall option) and violations of the ANSI C standard (pedantic option). The g option embeds debugging support in the executable file, which is saved in topo with the o topo option. Full optimization is enabled with the O3 option.

The warnings produced by the C compiler are sent to standard output. Here the first and last warnings result from the pedantic option; the other warnings result from the Wall option:

$ gcc -Wall -g -O3 -pedantic -o topo topo.c
In file included from topo.c:2:
/usr/include/ctype.h:65: warning: comma at end of enumerator list
topo.c:13: warning: return-type defaults to 'int'
topo.c: In function 'main':
topo.c:14: warning: unused variable 'c'
topo.c: In function 'getline':
topo.c:44: warning: 'c' might be used uninitialized in this function

When compiling programs that use the X11 include files and libraries, you may need to use the I and L options to tell gcc where to find those include files and libraries. The next example uses those options and instructs gcc to link the program with the basic X11 library:

$ gcc -I/usr/X11R6/include plot.c -L/usr/X11R6/lib -lX11

GetFileInfo: Displays file attributes

GetFileInfo [option] file

The GetFileInfo utility displays file attributes (page 95), including the file's type and creator code, creation and last modification time, and attribute flags such as the invisible and locked flags.

Arguments

The file is a single file that GetFileInfo displays information about.

Options

The options for GetFileInfo correspond to the options for SetFile (page 834).

Without an option, GetFileInfo reports on the metadata of file, indicating which flags are set, the file's type and creator codes, and its creation and modification dates. Missing data is omitted. When you specify an option, GetFileInfo displays the information specified by that option only. This utility accepts a single option; it silently ignores any additional options that you provide.

aflag
 | (attribute) Reports the status of a single attribute flag named flag. Displays 1 if flag is set and 0 if flag is not set. The flag must follow the a immediately, without any intervening SPACEs. See Table 4-3 on page 96 for a list of attribute flags.
c | (creator) Displays the creator code of file. If file has no creator code, this option displays an error message.
d | (date) Displays the creation date of file as mm/dd/yyyy hh:mm:ss, using a 24-hour clock.
m | (modification) Displays the modification date of file as mm/dd/yyyy hh:mm:ss, using a 24-hour clock.
t | (type) Displays the type code of file. If file has no type code, this option displays an error message.

Discussion

Without an option, GetFileInfo displays flags as the string avbstclinmedz, with uppercase letters denoting which flags are set. See page 95 for a discussion of attribute flags.

Notes

The GetFileInfo utility is available in /Developer/Tools when the Developer Tools package (page 478) is installed.

You can use the SetFile utility (page 834) to set file attributes. You can set UNIX permissions and ownership (page 87) by using chmod (page 676) and chown (page 682), and can display this information by using ls (page 777) or stat (page 851).

Directories do not have type or creator codes, and they may not have all flags. The GetFileInfo utility cannot read special files such as device files.

Examples

The first example shows the output from GetFileInfo when you call it without an option. Because this utility is not in the default PATH (page 285), you must either change your PATH variable or call it with an absolute pathname, as shown following:

$ /Developer/Tools/GetFileInfo picture.jpg
file: "/private/tmp/picture.jpg"
type: "JPEG"
creator: "GKON"
attributes: avbstClinmedz
created: 07/18/2005 15:15:26
modified: 07/18/2005 15:15:26

The only uppercase letter on the attributes line is C, indicating that this flag is set. The c flag tells the Finder to look for a custom icon for this file. See Table 4-3 on page 96 for a list of flags.

The next example uses the a flag to display the attribute flags for a file:

$ /Developer/Tools/GetFileInfo -a /Applications/Games/Alchemy/Alchemy
avBstclInmedz

The output shows that the b and i flags are set.

The GetFileInfo utility can process only one file per invocation. The following multiline bash command uses a for loop (page 537) to display the creator codes of multiple files. The echo command displays the name of the file being examined because GetFileInfo does not always display the name of the file:

$ for i in *
> do echo -n "$i: "; /Developer/Tools/GetFileInfo -c "$i"
> done
Desktop: Desktop is a directory and has no creator
Documents: Documents is a directory and has no creator
...
aa: ""
ab: ""
...

grep: Searches for a pattern in files

grep [options] pattern [file-list]

The grep utility searches one or more files, line by line, for a pattern, which can be a simple string or another form of a regular expression. It takes various actions, as specified by its options, each time it finds a line that contains a match for the pattern. This utility takes its input from files you specify on the command line or from standard input.

Arguments

The pattern is a regular expression, as defined in Appendix A. You must quote regular expressions that contain special characters, SPACEs, or TABs. An easy way to quote these characters is to enclose the entire expression within single quotation marks.

The file-list specifies the pathnames of one or more ordinary files that grep searches. With the r option, file-list may contain directories whose contents are searched.

Options

Without any options, grep sends lines that contain a match for pattern to standard output. When you specify more than one file on the command line, grep precedes each line that it displays with the name of the file that it came from followed by a colon.

Major Options

You can use only one of the following three options at a time. Normally you do not need to use any, because grep defaults to G, which is regular grep.

E | (extended) Interprets pattern as an extended regular expression (page 905). The command grep E is the same as egrep. See "Notes" later in this section.
F | (fixed) Interprets pattern as a fixed string of characters. The command grep F is the same as fgrep.
G | (grep) Interprets pattern as a basic regular expression. It is the default major option if none is specified.

Other Options

Accepts the common options described on page 663.

count | c | Displays only the number of lines that contain a match in each file.
context=n | C | n
 |  | Displays n lines of context around each matching line.
file=file | f | file
 |  | Reads file, which contains one pattern per line, and finds lines in the input that match a pattern in file.
no-filename | h | (header) Does not display the filename at the beginning of each line when searching through multiple files.
ignore-case | i | Causes lowercase letters in the pattern to match uppercase letters in the file, and vice versa. Use this option when you are searching for a word that may be at the beginning of a sentence (that is, the word may or may not start with an uppercase letter).
files-with-matches |  |
 | l | (list, lowercase "l") Displays only the name of each file that contains one or more matches. A filename is displayed just once, even if the file contains more than one match.
max-count=n | mn
 |  | Stops reading each file, or standard input, after displaying n lines containing matches.
line-number | n | Precedes each line by its line number in the file. The file does not need to contain line numbers.
quiet or silent | q | Does not write anything to standard output; just sets the exit code.
recursive | r or R
 |  | Recursively descends directories in file-list and processes files within these directories.
no-messages | s | (silent) Does not display an error message if a file in file-list does not exist or is not readable.
invert-match | v | Causes lines not containing a match to satisfy the search. When you use this option by itself, grep displays all lines that do not contain a match for the pattern.
word-regexp | w | With this option, the pattern must match a whole word. This option is helpful if you are searching for a specific word that may also appear as a substring within another word in the file.
line-regexp | x | The pattern matches whole lines only.

Notes

The grep utility returns an exit status of 0 if it finds a match, 1 if it does not find a match, and 2 if the file is not accessible or a syntax error occurs.

egrep and fgrep

Two utilities perform functions similar to that of grep. The egrep utility (same as grep E) allows you to use extended regular expressions (page 905), which include a different set of special characters than basic regular expressions do (page 903). The fgrep utility (same as grep F) is fast and compact but processes only simple strings, not regular expressions.

Examples

The following examples assume that the working directory contains three filestesta, testb, and testc:

File testa   File testb   File testc

aaabb        aaaaa        AAAAA
bbbcc        bbbbb        BBBBB
ff-ff        ccccc        CCCCC
cccdd        ddddd        DDDDD
dddaa

The grep utility can search for a pattern that is a simple string of characters. The following command line searches testa and displays each line containing the string bb:

$ grep bb testa
aaabb
bbbcc

The v option reverses the sense of the test. The following example displays the lines in testa without bb:

$ grep -v bb testa
ff-ff
cccdd
dddaa

The n option displays the line number of each displayed line:

$ grep -n bb testa
1:aaabb
2:bbbcc

The grep utility can search through more than one file. Here grep searches through each file in the working directory. The name of the file containing the string precedes each line of output.

$ grep bb *
testa:aaabb
testa:bbbcc
testb:bbbbb

When the search for the string bb is done with the w option, grep produces no output because none of the files contains the string bb as a separate word:

$ grep -w bb *
$

The search that grep performs is case sensitive. Because the previous examples specified lowercase bb, grep did not find the uppercase string BBBBB in testc. The i option causes both uppercase and lowercase letters to match either case of letter in the pattern:

$ grep -i bb *
testa:aaabb
testa:bbbcc
testb:bbbbb
testc:BBBBB
$ grep -i BB *
testa:aaabb
testa:bbbcc
testb:bbbbb
testc:BBBBB

The c option displays the number of lines in each file that contain a match:

$ grep -c bb *
testa:2
testb:1
testc:0

The f option finds matches for each pattern in a file of patterns. The next example shows gfile, which holds two patterns, one per line. The grep utility searches for matches to the patterns in gfile:

$ cat gfile
aaa
bbb
$ grep -f gfile test*
testa:aaabb
testa:bbbcc
testb:aaaaa
testb:bbbbb

The following command line displays from the file text2 those lines that contain a string of characters starting with st, followed by zero or more characters (.* represents zero or more characters in a regular expressionsee Appendix A), and ending in ing:

$ grep 'st.* ing' text2
...

The ^ regular expression, which matches the beginning of a line, can be used alone to match every line in a file. Together with the n option, ^ can be used to display the lines in a file, preceded by their line numbers:

$ grep -n '^' testa
1:aaabb
2:bbbcc
3:ff-ff
4:cccdd
5:dddaa

The next command line counts the number of times #include statements appear in C source files in the working directory. The h option causes grep to suppress the filenames from its output. The input to sort consists of all lines from *.c that match #include. The output from sort consists of an ordered list of lines that contains many duplicates. When uniq with the c option processes this sorted list, it outputs repeated lines only once, along with a count of the number of repetitions in its input.

$ grep -h '#include' * .c | sort | uniq -c
9 #include "buff.h"
2 #include "poly.h"
1 #include "screen.h"
6 #include "window.h"
2 #include "x2.h"
2 #include "x3.h"
2 #include <math.h>
3 #include <stdio.h>

The final command calls the vim editor with a list of files in the working directory that contain the string Sampson. The $(...) command substitution construct (page 327) causes the shell to execute grep and supply vim with a list of filenames that you want to edit:

$ vim $(grep -l 'Sampson' *)
...

The single quotation marks are not necessary in this example, but they are required if the regular expression you are searching for contains special characters or SPACEs. It is a good habit to quote the pattern so that the shell does not interpret any special characters it may contain.

gzip: Compresses or decompresses files

gzip [options] [file-list]

gunzip [options] [file-list]

zcat [file-list]

The gzip utility compresses files; gunzip restores files compressed with gzip; zcat displays files compressed with gzip.

Arguments

The file-list specifies the pathnames of one or more files that are to be compressed or decompressed. If a directory appears in file-list with no recursive option, gzip/gunzip issues an error message and ignores the directory. With the recursive option, gzip/gunzip recursively compresses/decompresses files within the directory hierarchy.

Without a file-list, or with the special option , gzip reads from standard input. The stdout option causes gzip and gunzip to write to standard output.

The information in this section also applies to gunzip, which is a link to gzip.

Options

Accepts the common options described on page 663.

stdout | c | Writes the results of compression or decompression to standard output instead of overwriting the original file.
decompress or uncompress | d | Decompresses a file compressed with gzip. This option with gzip is equivalent to the gunzip command.
fast or best | n | Controls the tradeoff between the speed of compression and the amount of compression. The n is a digit from 1 to 9. Level 1 is the fastest (least) compression, and level 9 is the best (slowest and most) compression. The default level is 6. The options fast and best are synonyms for 1 and 9, respectively.
force | f | Overwrites an existing output file.
list | l | For each compressed file in file-list, displays the file's compressed and decompressed sizes, the compression ratio, and the name of the file before compression. Use with verbose to display additional information.
quiet | q | Suppresses warning messages.
recursive | r | Recursively descends directories in file-list and compresses/decompresses files within these directories.
test | t | Verifies the integrity of a compressed file. Displays nothing if the file is OK.
verbose | v | Displays the name of the file, the name of the compressed file, and the amount of compression as each file is processed.

Discussion

Compressing files reduces disk space requirements and the time needed to transmit files between systems. When gzip compresses a file, it adds the extension .gz to the filename. For example, compressing the file fname creates the file fname.gz and deletes the original file. To restore fname, use the command gunzip with the argument fname.gz.

Almost all files become much smaller when compressed with gzip. On rare occasions a file will grow in size, but only by a relatively small amount. The type of a file and its contents (as well as the n option) determine how much smaller a file becomes; the sizes of text files are often reduced by 60 to 70 percent.

The attributes of a file, such as its owner, permissions, and modification and access times, are left intact when gzip compresses and gunzip decompresses a file.

If the compressed version of a file already exists, gzip reports that fact and asks for your confirmation before overwriting the existing file. If a file has multiple links to it, gzip issues an error message and exits. The force option overrides the default behavior in both of these situations.

Notes

For more information refer to "gzip: Compresses a File" on page 55.

In addition to the gzip format, gunzip recognizes several other compression formats. For example, it can decompress files that have been compressed with compress.

To see an example of a file that becomes larger when compressed with gzip, compare the size of a file that has been compressed once with the same file compressed with gzip again. Because gzip complains when you give it an argument with the extension .gz, you need to rename the file before compressing it a second time.

The tar utility with the z modifier (page 864) calls gzip.

The following related utilities display and manipulate compressed files. None of these utilities changes the files that it works on.

zcat file-list | Works like cat except that it uses gunzip to decompress file-list as it copies files to standard output.
zdiff [options] file1 [file2]
 | Works like diff (page 707) except that file1 and file2 are decompressed with gunzip as needed. The zdiff utility accepts the same options as diff. If you omit file2, zdiff compares file1 with the compressed version of file1.
zless file-list | Works like less except that it uses gunzip to decompress file-list as it displays files.

Examples

In the first example, gzip compresses two files. Then gunzip decompresses one of the files. When a file is compressed and decompressed, its size changes but its modification time remains the same.

$ ls -l
total 175
-rw-rw-r-- 1 alex alex 33557 Jul 20 17:32 patch-2.0.7
-rw-rw-r-- 1 alex alex 143258 Jul 20 17:32 patch-2.0.8
$ gzip *
$ ls -l
total 51
-rw-rw-r-- 1 alex alex 9693 Jul 20 17:32 patch-2.0.7.gz
-rw-rw-r-- 1 alex alex 40426 Jul 20 17:32 patch-2.0.8.gz
$ gunzip patch-2.0.7.gz
$ ls -l
total 75
-rw-rw-r-- 1 alex alex 33557 Jul 20 17:32 patch-2.0.7
-rw-rw-r-- 1 alex alex 40426 Jul 20 17:32 patch-2.0.8.gz

In the next example, the files in Jenny's home directory are archived using the cpio utility (page 809). The archive is compressed with gzip before it is written to the archive file.

$ find /Users/jenny -depth -print | cpio -oBm | gzip > /Volumes/Backups/jenny.cpio.gz

head: Displays the beginning of a file

head [options] [file-list]

The head utility displays the beginning (head) of a file. This utility takes its input from one or more files you specify on the command line or from standard input.

Arguments

The file-list specifies the pathnames of one or more files that head displays. When you specify more than one file, head displays the filename before displaying the first few lines of each file. When you do not specify a file, head takes its input from standard input.

Options

n n |
 | (lines) Displays the first n lines of a file. You can use n to specify n lines without using the n option. The n must be a positive number.

Notes

The head utility displays the first ten lines of a file by default.

Examples

The examples in this section are based on the following file:

$ cat eleven
line one
line two
line three
line four
line five
line six
line seven
line eight
line nine
line ten
line eleven

Without any arguments, head displays the first ten lines of a file:

$ head eleven
line one
line two
line three
line four
line five
line six
line seven
line eight
line nine
line ten

The next example displays the first three lines (n 3) of the file:

$ head -n 3 eleven
line one
line two
line three

The following example is equivalent to the preceding one:

$ head -3 eleven
line one
line two
line three

kill: Terminates a process by PID number

kill [option] PID-list

The kill utility terminates one or more processes by sending them signals. For kill to work, the process must belong to the user executing kill, with one exception: Superuser can terminate any process.

Arguments

The PID-list specifies process identification (PID) numbers of processes that kill is to terminate.

Options

You can specify a signal number or name, preceded by a hyphen, as an option to cause kill to send the signal you specify to the PID-list. By default, kill sends a software termination signal (SIGTERM, signal number 15).

l (list) Displays a list of signals. (Do not specify PID-list.)

Notes

See also killall on page 763.

See Table 13-5 on page 577 for a list of signals. The command kill l displays a complete list of signal numbers and names.

In addition to the kill utility, kill is a builtin in the Bourne Again and TC Shells. The builtins work similarly to the utility described here. Give the command /bin/kill to use the kill utility and the command kill to use the builtin. It does not usually matter which one you use.

The shell displays the PID number of a background process when you initiate the process. You can also use the ps utility to determine PID numbers.

If the software termination signal does not terminate a process, try using a KILL signal (signal number 9). A process can choose to ignore any signal except KILL.

The kill utility/builtin accepts job identifiers in place of the PID-list. Job identifiers consist of a percent sign (%) followed by either a job number (page 131) or a string that uniquely identifies the job.

To terminate all processes that the current login process initiated and have the operating system log you out, give the command kill 9 0.

Examples

The first example shows a command line executing the file compute as a background process and the kill utility terminating it:

$ compute &
[2] 259
$ kill 259
$ RETURN
[2]+   Terminated                 compute

The next example shows the ps utility determining the PID number of the background process running a program named robots and the kill utility terminating robots with the TERM signal:

$ ps
  PID  TT  STAT        TIME COMMAND
 5382  p1  Ss       0:00.27 -bash
18957  p1  S+       0:00.01 robots
14863  p2  Ss       0:00.25 -bash
$ kill -TERM 18957
$

killall: Terminates a process by name

killall [option] name-list

The killall utility sends signals to processes executing specified commands. For killall to work, the processes must belong to the user executing killall, with one exception: Superuser can terminate any process.

Arguments

The name-list is a SPACE-separated list of names of programs that are to receive signals.

Options

You can specify a signal number or name, preceded by a hyphen, as an option to cause killall to send the signal you specify to name-list. By default killall sends software termination signals (signal number 15, SIGTERM).

l (list) Displays a list of signals (but kill l displays a better list). (Do not specify a name-list.)

Notes

See also kill on page 761.

See Table 13-5 on page 577 for a list of signals. The command kill l displays a complete list of signal numbers and names.

If the software termination signal does not terminate a process, try using a KILL signal (signal number 9). A process can choose to ignore any signal except KILL.

You can use the ps utility (page 819) to determine the name of the program you want to terminate.

Examples

You can give the following commands to experiment with killall:

$ sleep 60 &
[1] 23274
$ sleep 50 &
[2] 23275
$ sleep 40 &
[3] 23276
$ sleep 120 &
[4] 23277
$ killall sleep
[1]   Terminated             sleep 60
[2]   Terminated             sleep 50
[3]-  Terminated             sleep 40
[4]+  Terminated             sleep 120

If you want to terminate all instances of the Safari browser, give the following command to determine how Safari was called:

$ ps ax | grep -i safari
20316  ?? S   0:04.38 /Applications/Safari.app/Contents/MacOS/Safari -psn_0
24216  p1 R+  0:00.00 grep -i safari

The next command, run as root, terminates all instances of the Safari browser:

# killall Safari

launchctl: Controls the launchd daemon

launchctl [subcommand [options] [arguments]]

The launchctl utility controls the launchd daemon (page 456).

Arguments

The subcommand is the command that launchctl sends to launchd. Table VI-15 lists the subcommands and the options and arguments that each accepts. Without a subcommand, launchctl reads subcommands, options, and arguments from standard input, one set per line. Without a subcommand, when standard input comes from the keyboard, launchctl prompts you for input.

Table VI-15. launchctl subcommands

Subcommand

Argument

Description

help

None

Displays a help message

list

None

Lists jobs loaded into launchd

load [w]

Job configuration file

Loads the job named by the argument

reloadttys

None

Rereads the /etc/ttys file

shutdown

None

Prepares for shutdown by removing all jobs

start

Job name

Starts the job named by the argument

stop

Job name

Stops the job named by the argument

unload [w]

Job configuration file

Unloads the job named by the argument

Options

Only the load and unload subcommands take options.

w

(write) When loading a file, remove the Disabled key and save the modified configuration file. When unloading a file, add the Disabled key and save the modified configuration file.

Discussion

The launchd daemon (page 456) manages system daemons and background tasks, called jobs. Each job is described by a job configuration file, which is a property list file (page 453) in the format defined by the launchd.plist man page. The launchctl utility is the user interface to launchd.

For security reasons, nonroot users cannot communicate with the system's primary launchd process, which is PID 1. When a nonroot user loads jobs, OS X creates a new instance of launchd for that user. When all of its jobs are unloaded, that instance of launchd quits.

Notes

The launchctl utility and launchd daemon were introduced in Mac OS X version 10.4. Under version 10.3 and earlier, system jobs are managed by init, xinetd, and cron.

For more information refer to "launchctl: Controls launchd" on page 458.

Examples

The first example, run as root, uses the list subcommand to list launchd jobs running on the local system:

# launchctl list
com.apple.atprintd.i9900
com.apple.KernelEventAgent
com.apple.mDNSResponder
com.apple.nibindd
com.apple.periodic-daily
com.apple.periodic-monthly
com.apple.periodic-weekly
com.apple.portmap
com.apple.servermgrd
com.apple.syslogd
com.apple.watchdogtimerd
com.vix.cron
org.postfix.master
org.xinetd.xinetd
com.openssh.sshd
com.apple.cups-lpd

Next root enables the ntalk service. Looking at the ntalk.plist file before and after the launchctl command shows that launchctl has modified the file by removing the Disabled key.

# cat /System/Library/LaunchDaemons/ntalk.plist
...
<dict>
        <key>Disabled</key>
        <true/>
        <key>Label</key>
        <string>com.apple.ntalkd</string>
...
# launchctl load -w /System/Library/LaunchDaemons/ntalk.plist
# cat /System/Library/LaunchDaemons/ntalk.plist
...
<dict>
        <key>Label</key>
        <string>com.apple.ntalkd</string>
...

Without any arguments, launchctl prompts for subcommands on standard input. Use CONTROL-D to exit from launchctl. In the last example, root causes launchctl to display a list of jobs and then to stop the job that would launch cron if there were any crontab files (page 697):

# launchctl
launchd% list
com.apple.KernelEventAgent
com.apple.mDNSResponder
com.apple.nibindd
com.apple.periodic-daily
com.apple.periodic-monthly
com.apple.periodic-weekly
com.apple.portmap
com.apple.syslogd
com.vix.cron
org.postfix.master
org.xinetd.xinetd
com.openssh.sshd
launchd% stop com.vix.cron
launchd% CONTROL-D
#

less: Displays text files, one screen at a time

less [options] [file-list]

The less utility displays text files, one screen at a time.

Arguments

The file-list specifies the pathnames of one or more files that less is to display. Without a file-list, less reads from standard input.

Options

Accepts the common options described on page 663.

clear-screen | c | Repaints the screen from the top line down instead of scrolling.
quit-at-eof | e | (exit) Normally less requires you to enter q to terminate the utility. This option exits automatically the second time less reads the end of file.
QUIT-AT-EOF | E | (exit) Similar to e, except that less exits automatically the first time it reads the end of file.
quit-if-one-screen |  |
 | F | Displays the file and quits if the file can be displayed on a single screen.
ignore-case | i | Causes a search for a string of lowercase letters to match both uppercase and lowercase letters (see /regular-expression on page 770). This option is ignored if you specify a pattern that includes uppercase letters.
IGNORE-CASE |  |
 | I | Causes a search for a string of letters of any case to match both uppercase and lowercase letters, regardless of the case of the search pattern (see /regular-expression on page 770).
long-prompt | m | Reports the percentage of the file that you have viewed with each prompt. This option reports byte numbers when less reads from standard input because less has no way of determining how much input there is.
LINE-NUMBERS |  |
 | N | Displays a line number at the start of each line.
 | Psprompt
 |  | (prompt string) Changes the short prompt string (the prompt that appears at the bottom of each screen of output) to prompt. Enclose prompt in quotation marks if it contains SPACEs. You can use special symbols in prompt that less will replace with other values when it displays the prompt. For example, less displays the current filename in place of %f. See the less man page for a list of these special symbols and descriptions of other prompts. Custom prompts are useful if you are running less from within another program and want to give instructions or information to the user of that program. The default prompt is the name of the file displayed in reverse video.
squeeze-blank-lines |  |
 | s | Displays multiple, adjacent blank lines as a single blank line. When you use less to display text that has been formatted for printing with blank space at the top and bottom of each page, this option shortens these headers and footers to a single line.
tabs=n | xn |
 |  | Sets TAB stops n characters apart. The default is eight characters.
window=n | [z]n
 |  | Sets the scrolling size to n lines. The default is the size of the display. Each time you move forward or backward a page, you move n lines. See "Commands."
 | +command
 |  | Any command you can give less while it is running can also be given as an option by preceding it with a plus sign (+) on the command line. See the "Commands" section. A command preceded by a plus sign on the command line is executed as soon as less starts and applies only to the first file.
 | ++command
 |  | Similar to +command except that command is applied to every file in file-list, not just the first one.

Notes

The phrase "less is more" explains the origin of this utility. The more utility is the original Berkeley UNIX pager. The less utility is similar to more but includes many enhancements. (Under OS X, less and more are two links to the same file.) After displaying a screen of text, less displays a prompt and waits for you to enter a command. You can skip forward and backward in the file, invoke an editor, search for a pattern, or perform any number of other tasks.

See the v command in the next section for information on how you can edit the file that you are viewing with less.

You can set the options to less either from the command line when you call less or by setting the LESS environment variable. For example, you can use the following command from bash to use less with the x4 and s options:

$ export LESS="-x4 -s"

Normally you would set LESS in ~/.bash_profile if you are using bash or in ~/.login if you are using tcsh. Once you have set the LESS variable, less is invoked with the specified options each time you call it. Any options you give on the command line override the settings in the LESS variable. The LESS variable is used both when you call less from the command line and when less is invoked by another program, such as man. To specify less as the pager to use with man and other programs, set the environment variable PAGER to less. For example, with bash you can add the following line to ~/.bash_profile:

export PAGER=less

Commands

Whenever less pauses, you can enter any of a large number of commands. This section describes some commonly used commands. Refer to the less man page for the complete list of commands. The optional numeric argument n defaults to 1, with the exceptions noted. You do not need to press RETURN to terminate these commands.

nb or n CONTROL-B | (backward) Scrolls backward n lines. The default value of n is the size of the screen.
nd or n CONTROL-D | (down) Scrolls forward n lines. The default value of n is one-half the screen size. When you specify n, it becomes the new default value for this command.
F | (forward) Scrolls forward. If the end of the input is reached, this command waits for more input and then continues scrolling. It allows you to use less in a manner similar to tail f (page 859), except that less paginates the output as it appears.
ng | (go) Goes to line number n. This command may not work if the file is read from standard input and you have moved too far into the file. The default value of n is 1.
h or H | (help) Displays a summary of all available commands. The summary is displayed using less, as the list of commands is quite long.
n RETURN or nj | (jump) Scrolls forward n lines. The default value of n is 1.
q or :q | Terminates less.
nu or n CONTROL-U | (up) Scrolls backward n lines. The default value of n is one-half the screen size. When you specify n, it becomes the default value for this command.
v | Brings the current file into an editor with the cursor on the current line. The less utility uses the editor specified in the EDITOR environment variable. If EDITOR is not set, less uses vi (which is typically linked to vim).
nw | Scrolls backward like nb, except that the value of n becomes the new default value for this command.
ny or nk | Scrolls backward n lines. The default value of n is 1.
nz | Displays the next n lines like n SPACE except that the value of n, if present, becomes the new default value for the z and SPACE commands.
nSPACE | Displays the next n lines. Pressing SPACE by itself displays the next screen of text.
/regular-expression
 | Skips forward in the file, looking for lines that contain a match for regular-expression. If you begin regular-expression with an exclamation point (!), this command looks for lines that do not contain a match for regular-expression. If regular-expression begins with an asterisk (*), this command continues the search through file-list. (A normal search stops at the end of the current file.) If regular-expression begins with an at sign (@), this command begins the search at the start of file-list and continues to the end of file-list.
?regular-expression
 | This command is similar to the previous one but searches backward through the file (and file-list). Beginning regular-expression with an asterisk (*) causes the search to continue backward through file-list to the beginning of the first file. Beginning regular-expression with an at sign (@) causes the search to start with the last line of the last file in file-list and progress toward the first line of the first file.
{ or ( or [ | If one of these characters appears in the top line of the display, this command scrolls forward to the matching right brace, parenthesis, or bracket. For example, typing { causes less to move the cursor forward to the matching }.
} or ) or ] | Similar to the preceding commands, these commands move the cursor backward to the matching left brace, parenthesis, or bracket.
CONTROL-L | Redraws the screen. This command is useful if the text on the screen has become garbled.
[n]:n | Skips to the next file in file-list. With n, skips to the nth next file in file-list.
![command line] | Executes command line under the shell specified by the SHELL environment variable, or sh (a copy of bash) by default. A percent sign (%) in command line is replaced by the name of the current file. If you omit command line, less starts an interactive shell.

Examples

The following example displays the file memo.txt. To see more of the file, the user presses the SPACE bar in response to the less prompt at the bottom left of the screen:

$ less memo.txt
...
memo.txt SPACE
...

In the next example, the user changes the prompt to a more meaningful message and uses the N option to display line numbers. The user then instructs less to skip forward to the first line containing the string procedure.

$ less -Ps"Press SPACE to continue, q to quit" -N +/procedure ncut.icn
    28 procedure main(args)
    29     local filelist, arg, fields, delim
    30
    31     filelist:=[]
...
    45     # Check for real field list
    46     #
    47     if /fields then stop("-fFIELD_LIST is required.")
    48
    49     # Process the files and output the fields
Press SPACE to continue, q to quit

ln: Makes a link to a file

ln [options] existing-file [new-link]

ln [options] existing-file-list directory

The ln utility creates hard or symbolic links to one or more files. You can create a symbolic linkbut not a hard linkto a directory.

Arguments

In the first format the existing-file is the pathname of the file that ln is to create a link to. The new-link is the pathname of the new link. When you are creating a symbolic link, the existing-file can be a directory. If you omit new-link, ln creates a link to existing-file in the working directory and uses the same simple filename as existing-file.

In the second format the existing-file-list specifies the pathnames of the ordinary files that ln is to create links to. The ln utility establishes the new links in the directory. The simple filenames of the entries in the directory are the same as the simple filenames of the files in the existing-file-list.

Options

f | (force) Normally ln does not create the link if new-link already exists. This option removes new-link before creating the link.
h | (no dereference) If new-link or directory is a symbolic link, do not follow it. Used in conjunction with f, this option replaces an existing symbolic link.
i | (interactive) If new-link already exists, this option prompts you before removing new-link. If you enter y or yes, ln removes new-link before creating the link. If you answer n or no, no new link is made.
s | (symbolic) Creates a symbolic link. When you use this option, the existing-file and new-link may be directories and may reside on different filesystems. Refer to "Symbolic Links" on page 105.

Notes

For more information refer to "Links" on page 102. The ls utility with the l option displays the number of hard links to a file (Figure 4-11, page 88).

Hard links

By default ln creates hard links. A hard link to a file is indistinguishable from the original file. All hard links to a file must be in the same filesystem. For more information refer to "Hard Links" on page 103.

Symbolic links

You can also use ln to create symbolic links. Unlike a hard link, a symbolic link can exist in a different filesystem from the linked-to file. Also, a symbolic link can point to a directory. For more information refer to "Symbolic Links" on page 105.

If new-link is the name of an existing file, ln does not create the link unless you use the f option or answer yes when using the i option.

Examples

The following command creates a link between the memo2 file in the /Users/zach/literature directory and the working directory. The file appears as memo2 (the simple filename of the existing file) in the working directory:

$ ln /Users/zach/literature/memo2 .

You can omit the period that represents the working directory from the preceding command. When you give a single argument to ln, it creates a link in the working directory.

The next command creates a link to the same file. This time the file appears as new_memo in the working directory:

$ ln /Users/zach/literature/memo2 new_memo

The following command creates a link that causes the same file to appear in another user's directory:

$ ln /Users/zach/literature/memo2 /Users/jenny/new_memo

You must have write and execute access permissions to the other user's directory for this command to work. If you own the file, you can use chmod to give the other user write access permission to the file.

The next command creates a symbolic link to a directory. The ls ld command shows the link:

$ ln -s /usr/local/bin bin
$ ls -ld bin
lrwxr-xr-x 1 zach zach 14 Feb 10 13:26 bin -> /usr/local/bin

The final example attempts to create a symbolic link named memo1 to the file memo2. Because the file memo1 already exists, ln refuses to make the link. When you use the i option, ln asks whether you want to replace the existing memo1 file with the symbolic link. If you enter y, ln removes the old memo1 file and creates the link.

$ ls -l memo?
-rw-rw-r--  1 zach group 224 Jul 31 14:48 memo1
-rw-rw-r--  1 zach group 753 Jul 31 14:49 memo2
$ ln -s memo2 memo1
ln: memo1: File exists
$ ln -s -i memo2 memo1
ln: replace 'memo1'? y
$ ls -l memo?
lrwxr-xr-x  1 zach group 5 Jul 31 14:49 memo1 -> memo2
-rw-rw-r--  1 zach group 753 Jul 31 14:49 memo2

You can also use the f option to cause ln to overwrite a file.

lpr: Sends files to printers

lpr [options] [file-list]

lpq [options] [job-identifier-list]

lprm [options] [job-identifier-list]

The lpr utility places one or more files into a print queue, providing orderly access to printers for several users or processes. This utility can work with printers attached to remote systems. You can use the lprm utility to remove files from the print queues and the lpq utility to check the status of files in the queues.

Arguments

The file-list specifies the pathnames of one or more files that lpr is to print. Often these files are text files, but many systems are configured so that lpr can print a variety of file types. Without a file-list, lpr accepts input from standard input.

The job-identifier-list is a list of job numbers or usernames. If you do not know the job number of a print job, use lpq to display a list of print jobs.

Options

Some of the following options depend on what type of file is being printed as well as on how the system is configured for printing.

l | (lowercase "l") Specifies that lpr should not preprocess (filter) the file being printed. Use this option when the file is already formatted for the printer.
P | printer
 | Sends the print jobs to the queue for the printer named printer. If you do not use this option, print jobs are sent to the default printer for the local system. The acceptable values for printer are found in the file /etc/printcap and vary from system to system.
r | (remove) Deletes the files in file-list after calling lpr.
# | n
 | Prints n copies of each file.

Discussion

The lpr utility takes input from files you specify on the command line or from standard input and adds the files to the print queue as print jobs. It also assigns a unique identification number to each print job. The lpq utility displays the job numbers of the print jobs that lpr has set up; you can use the lprm utility to remove a print job from the print queue.

lpq

The lpq utility displays information about jobs in a print queue. Without an argument, lpq lists the print jobs queued for the default printer. Use lpr's P printer option with lpq to look at other print queueseven those for printers connected to other systems. With the l (lowercase "l") option, lpq displays more information about each job. If you supply a username as an argument, lpq displays only the print jobs belonging to that user.

lprm

One item displayed by lpq is the job number for each print job in the queue. To remove a job from the print queue, supply the job number as an argument to lprm. Unless you are Superuser, you can remove only your own print jobs. Even as Superuser you may not be able to remove a job from a queue for a remote printer. If you do not give any arguments to lprm, it removes the currently active print job (that is, the job that is now printing) from the queue, if you own that job.

Notes

If you normally use a printer other than the system default printer, you can set up lpr to use another printer as your personal default by assigning the name of this printer to the environment variable PRINTER. For example, if you use bash, you can add the following line to ~/.bash_profile to set your default printer to the printer named ps:

export PRINTER=ps

lpstat

Printers configured through the Mac OS X GUI are accessible through lpr. The command lpstat t displays the names of the printers you can use.

LPD and LPR

Traditionally UNIX had two printing systems: the BSD Line Printer Daemon (LPD) and the System V Line Printer system (LPR). Both of these systems have undergone modifications and replacements. A newer system, CUPS, is the default printing system under Mac OS X.

CUPS

CUPS (Common UNIX Printing System) is a cross-platform print server built around IPP (Internet Printing Protocol), which is based on HTTP. CUPS provides a number of printer drivers and can print different types of files, including PostScript files. CUPS provides System V and BSD command line interfaces and, in addition to IPP, supports the LPD/LPR, HTTP, SMB, and JetDirect (socket) protocols, among others.

This section describes the LPD command line interface that runs under CUPS.

Examples

The first command sends the file named memo2 to the default printer:

$ lpr memo2

Next a pipe sends the output of ls to the printer named deskjet:

$ ls | lpr -Pdeskjet

The next example paginates and sends the file memo to the printer:

$ pr -h "Today's memo" memo | lpr

The next example shows a number of print jobs queued for the default printer. Alex owns all of these jobs as well as the one that is being printed (the active job). Jobs 635 and 639 were created by sending input to lpr's standard input; job 638 was created by giving ncut.icn as an argument to the lpr command. The last column shows the size of each print job.

$ lpq
deskjet is ready and printing
Rank   Owner      Job Files                                Total Size
active alex       635 (stdin)                             38128 bytes
1st    alex       638 ncut.icn                             3587 bytes
2nd    alex       639 (stdin)                              3960 bytes

The final command removes job 638 from the default print queue:

$ lprm 638

ls: Displays information about one or more files

ls [options] [file-list]

The ls utility displays information about one or more files. It lists the information alphabetically by filename unless you use an option to change the order.

Arguments

When you do not provide an argument, ls displays the names of the visible files in the working directory (those files whose filenames do not begin with a period).

The file-list is a list of one or more pathnames of ordinary, directory, or device files. It can include ambiguous file references.

When you specify a directory, ls displays the contents of the directory. It displays the name of the directory only when needed to avoid ambiguity, such as when the listing includes more than one directory. When you specify an ordinary file, ls displays information about that one file.

Options

The options determine the type of information ls displays as well as the manner and order in which it displays the information. Without an option, ls displays a short list that contains only the names of files.

a | (all) Includes invisible files (those whose filenames begin with a period) in the listing. Without this option, ls does not list information about invisible files unless you include the name of an invisible file in the file-list. The * ambiguous file reference does not match a leading period in a filename (see page 134).
A | (almost all) The same as a but does not list the . and .. entries (the default behavior when running as root).
b | (backslash) Displays nonprinting characters in a filename, using backslash escape sequences similar to those used in C language strings. Table VI-16 lists some of these escape sequences. Other nonprinting characters are displayed as a backslash followed by an octal number.

Table VI-16. Backslash escape sequences

Sequence

Meaning

\b

BACKSPACE

\n

NEWLINE

\r

RETURN

\t

HORIZONTAL TAB

\v

VERTICAL TAB

\\

BACKSLASH

d | (directory) Displays the names of directories without displaying their contents. Without an argument, this option displays information about the working directory. This option does not dereference symbolic links (it lists a link, not the directory the link points to).
e | (extended) Displays ACLs (page 97) in a long listing (with the l option).
F | (file type) Displays a slash (/) after a directory, an asterisk (*) after an executable file, an at sign (@) after a symbolic link, an equal sign (=) after a socket, and a vertical bar (|) after a FIFO.
h | (human readable) Displays sizes in K (kilobyte), M (megabyte), and G (gigabyte) blocks, as appropriate. Works with the l option only.
i | (inode) Displays the inode number of each file. With the l option, this option displays the inode number in column 1 and shifts all other items one column to the right.
l | (long, lowercase "l") Lists more information about each file. Use this option with h to make file sizes more easily readable and with e to display ACLs. See "Discussion" for more information.
L | (dereference) Lists information about the file referenced by a symbolic link rather than information about the link itself.
m | (stream) Displays filenames in a comma-separated list that fills the width of the screen.
o | (flags) Displays file flags (page 92) when used with the l option.
q | (question mark) Displays nonprinting characters in a filename as question marks. When output goes to the screen, this is the default behavior.
r | (reverse) Displays the list of filenames in reverse sorted order.
R | (recursive) Recursively lists subdirectories.
s | (size) Displays the number of blocks allocated to the file. The size precedes the filename. With the l option, this option displays the size in column 1 and shifts all other items one column to the right. Set BLOCKSIZE (page 663) to specify a block size (the default is 512 bytes).
t | (time) Displays files sorted by modification time instead of by filename.
x | Displays files sorted by lines (the default display is sorted by columns).
1 | (number 1) Displays files one per line. When the output does not go to the screen, this behavior is the default.

Discussion

The ls long, extended listing (le option) displays the eight columns shown in Figure 4-11 on page 88. The first column, which contains 11 characters, is divided as described in the following paragraphs. The first character describes the type of file, as shown in Table VI-17.

Table VI-17. First character in a long ls display

Character

Meaning

Ordinary

b

Block device

c

Character device

d

Directory

p

FIFO (named pipe)

l

Symbolic link

The next nine characters of the first column represent the access permissions associated with the file. They are divided into three sets of three characters each.

The first three characters represent the owner's access permissions. If the owner has read access permission for the file, r appears in the first character position. If the owner is not permitted to read the file, a hyphen appears in this position. The next two positions represent the owner's write and execute access permissions. If w appears in the second position, the owner is permitted to write to the file; if x appears in the third position, the owner is permitted to execute the file. An s in the third position indicates that the file has both setuid and execute permissions. An S indicates setuid without execute permission. A hyphen indicates that the owner does not have the access permission associated with the character position.

In a similar manner the second and third sets of three characters represent the access permissions of the group the file is associated with and the access permissions of other users, respectively. An s in the third position of the second set of characters indicates that the file has setgid permission with execute permission, and an S indicates setgid without execute permission.

The last character is t if the sticky bit (page 955) is set with execute permission and T if it is set without execute permission. Refer to chmod on page 676 for information on changing access permissions.

When called with the le options, ls displays a plus sign (+) in the second column if the file has an ACL. If the file does not have an ACL, the second column is empty. See page 97 for more information on ACLs.

The third column indicates the number of hard links to the file. Refer to page 102 for more information on links.

The fourth and fifth columns display the name of the owner of the file and the name of the group the file is associated with, respectively.

The sixth column indicates the size of the file in bytes or, if information about a device file is being displayed, the major and minor device numbers. In the case of a directory, this number is the size of the directory file, not the size of the files that are found within the directory. (Use du to display the sum of the sizes of all files in a directory.) Use the h option to display the size of files in kilobytes, megabytes, or gigabytes.

The last two columns display the date and time the file was last modified and the filename, respectively.

Notes

Refer to page 133 for examples of using ls with ambiguous file references.

Examples

The first command line shows the ls utility with the x option, which sorts the files horizontally. The ls utility displays an alphabetical list of the names of the files in the working directory:

$ ls -x
bin	    c           calendar
execute     letters     shell

The F option appends a slash (/) to files that are directories, an asterisk to files that are executable, and an at sign (@) to symbolic links:

$ ls -Fx
bin/      c/          calendar
execute*  letters/    shell@

Next the l (long) option displays a long list. The files are still in alphabetical order:

$ ls -l
total 8
drwxrwxr-x  2 jenny jenny   80  May 20 09:17 bin
drwxrwxr-x  2 jenny jenny  144  Mar 26 11:59 c
-rw-rw-r--  1 jenny jenny  104  May 28 11:44 calendar
-rwxrw-r--  1 jenny jenny   85  May  6 08:27 execute
drwxrwxr-x  2 jenny jenny   32  Mar  6 22:56 letters
drwxrwxr-x 16 jenny jenny 1296  Jun  6 17:33 shell

The a (all) option lists all files, including invisible ones:

$ ls -a
.      .profile   c           execute    shell
..     bin        calendar    letters

Combining the a and l options displays a long listing of all files, including invisible files, in the working directory. This list is still in alphabetical order:

$ ls -al
total 12
drwxrwxr-x  6 jenny jenny  480 Jun  6 17:42 .
drwxrwxr-t 26 root  admin  816 Jun  6 14:45 ..
-rw-rw-r--  1 jenny jenny  161 Jun  6 17:15 .profile
drwxrwxr-x  2 jenny jenny   80 May 20 09:17 bin
drwxrwxr-x  2 jenny jenny  144 Mar 26 11:59 c
-rw-rw-r--  1 jenny jenny  104 May 28 11:44 calendar
-rwxrw-r--  1 jenny jenny   85 May  6 08:27 execute
drwxrwxr-x  2 jenny jenny   32 Mar  6 22:56 letters
drwxrwxr-x 16 jenny jenny 1296 Jun  6 17:33 shell

When you add the r (reverse) option to the command line, ls produces a list in reverse alphabetical order:

$ ls -ral
total 12
drwxrwxr-x 16 jenny jenny 1296 Jun  6 17:33 shell
drwxrwxr-x  2 jenny jenny   32 Mar  6 22:56 letters
-rwxrw-r--  1 jenny jenny   85 May  6 08:27 execute
-rw-rw-r--  1 jenny jenny  104 May 28 11:44 calendar
drwxrwxr-x  2 jenny jenny  144 Mar 26 11:59 c
drwxrwxr-x  2 jenny jenny   80 May 20 09:17 bin
-rw-rw-r--  1 jenny jenny  161 Jun  6 17:15 .profile
drwxrwxr-t 26  root admin  816 Jun  6 14:45 ..
drwxrwxr-x  6 jenny jenny  480 Jun  6 17:42 .

Use the t and l options to list files so that the most recently modified file appears at the top of the list:

$ ls -tl
total 8
drwxrwxr-x 16 jenny jenny 1296 Jun  6 17:33 shell
-rw-rw-r--  1 jenny jenny  104 May 28 11:44 calendar
drwxrwxr-x  2 jenny jenny   80 May 20 09:17 bin
-rwxrw-r--  1 jenny jenny   85 May  6 08:27 execute
drwxrwxr-x  2 jenny jenny  144 Mar 26 11:59 c
drwxrwxr-x  2 jenny jenny   32 Mar  6 22:56 letters

Together the r and t options cause the file you modified least recently to appear at the top of the list:

$ ls -trl
total 8
drwxrwxr-x  2 jenny jenny   32 Mar  6 22:56 letters
drwxrwxr-x  2 jenny jenny  144 Mar 26 11:59 c
-rwxrw-r--  1 jenny jenny   85 May  6 08:27 execute
drwxrwxr-x  2 jenny jenny   80 May 20 09:17 bin
-rw-rw-r--  1 jenny jenny  104 May 28 11:44 calendar
drwxrwxr-x 16 jenny jenny 1296 Jun  6 17:33 shell

The next example shows ls with a directory filename as an argument. The ls utility lists the contents of the directory in alphabetical order:

$ ls bin
c      e     lsdir

To display information about the directory file itself, use the d (directory) option. This option lists information only about the directory:

$ ls -dl bin
drwxrwxr-x 2 jenny jenny 80 May 20 09:17 bin

The following command displays a list of all invisible filenames (those starting with a period) in your home directory. Using this command is a convenient way to list the startup files in your home directory:

$ ls -d ~/.*
/Users/sam/.                    /Users/sam/.bashrc
/Users/sam/..                   /Users/sam/.history
...

A plus sign (+) to the right of the permissions in a long listing denotes the presence of an ACL for a file:

$ ls -l memo
-rw-r--r-- + 1 alex   alex   19 Jul 19 21:59 memo

Use the le options to display an ACL:

$ ls -le memo
-rw-r--r-- + 1 alex   alex 19 Jul 19 21:59 memo
 0: user:jenny allow  read

See page 680 for more examples of using ls to display ACLs.

make: Keeps a set of programs current

make [options] [target-files] [arguments]

The GNU make utility keeps a set of executable programs current, based on differences in the modification times of the programs and the source files that each program is dependent on.

Arguments

The target-files refer to targets on dependency lines in the makefile. When you do not specify a target-file, make updates the target on the first dependency line in the makefile. Command line arguments of the form name=value set the variable name to value inside the makefile. See "Discussion" for more information.

Options

If you do not use the f option, make takes its input from a file named GNUmakefile, makefile, or Makefile (in that order) in the working directory. In this section, this input file is referred to as makefile. Many users prefer to use the name Makefile because it shows up earlier in directory listings.

C | directory
 | (change directory) Makes directory the working directory before starting.
d | (debug) Displays information about how make decides what to do.
f | file
 | (input file) Uses file as input instead of makefile.
j | n
 | (jobs) Runs up to n commands at the same time instead of the default of one command. Running multiple commands simultaneously is especially effective if you are running OS X on a multiprocessor system. If you omit n, make does not limit the number of simultaneous jobs.
k | (keep going) Continues with the next file from the list of target-files instead of quitting when a construction command fails.
n | (no execution) Displays, but does not execute, the commands that make would execute to bring the target-files up-to-date.
s | (silent) Does not display the names of the commands being executed.
t | (touch) Updates the modification times of target files but does not execute any construction commands (page 489). Refer to touch on page 877.

Discussion

The make utility bases its actions on the modification times of the programs and the source files that each program is dependent on. Each of the executable programs, or target-files, is dependent on one or more prerequisite files. The relationships between target-files and prerequisites are specified on dependency lines in a makefile. Construction commands follow the dependency line, specifying how make can update the target-files.

Documentation

Refer to page 489 for more information about make and makefiles. For additional information refer to www.gnu.org/software/make/manual/make.html and to the make info page.

Although the most common use of make is to build programs from source code, this general-purpose build utility is suitable for a wide range of applications. Anywhere you can define a set of dependencies to get from one state to another represents a candidate for using make.

Much of make's power derives from the features you can set up in a makefile. For example, you can define variables using the same syntax found in the Bourne Again Shell. Always define the variable SHELL in a makefile; set it to the pathname of the shell you want to use when running construction commands. To define the variable and assign it a value, place the following line near the top of a makefile:

SHELL=/bin/sh

Assigning the value /bin/sh to SHELL allows you to use a makefile on other computer systems. On OS X systems, /bin/sh is a copy of /bin/bash. The make utility uses the value of the environment variable SHELL if you do not set SHELL in a makefile. If SHELL does not hold the path of the shell you intended to use and if you do not set SHELL in a makefile, the construction commands may fail.

Following is a list of additional features associated with make:

  • You can run specific construction commands silently by preceding them with an @ sign. For example, the following lines display a short help message when you run the command make help:

    help:
    @echo "You can make the following:"
    @echo " "
    @echo "libbuf.a       -- the buffer library"
    @echo "Bufdisplay     -- display any-format buffer"
    @echo "Buf2ppm        -- convert buffer to pixmap"

    Without the @ signs in the preceding example, make would display each of the echo commands before executing it. This way of displaying a message works because no file is named help in the working directory. As a result make runs the construction commands in an attempt to build this file. Because the construction commands display messages but do not build the file help, you can run make help repeatedly with the same result.

  • You can cause make to ignore the exit status of a command by preceding the command with a hyphen (-). For example, the following line allows make to continue regardless of whether the call to /bin/rm is successful (the call to /bin/rm fails if libbuf.a does not exist):

    -/bin/rm libbuf.a

  • You can use special variables to refer to information that might change from one use of make to the next. Such information might include the files that need updating, the files that are newer than the target, or the files that match a pattern. For example, you can use the variable $? in a construction command to identify all prerequisite files that are newer than the target file. This variable allows you to print any files that have changed since the last time you printed files.

    list:      .list
.list:     Makefile buf.h xtbuff_ad.h buff.c buf_print.c xtbuff.c
pr $? | lpr
date >.list

  • The target list depends on which source files need to be printed. The construction command pr $? | lpr prints only those source files that are newer than the file .list. The line date > .list modifies the .list file so that it is newer than any of the source files. The next time you run the command make list, only the files that have been changed are printed.

  • You can include other makefiles as if they were part of the current makefile. The following line causes make to read Make.config and treat the contents of that file as though it were part of the current makefile, allowing you to collect information common to more than one makefile in a single place:

    include Make.config

Notes

The section "make: Keeps a Set of Programs Current" on page 489 provides more information about make.

Examples

The first example causes make to bring the target-file named analysis up-to-date by issuing three cc commands. It uses a makefile named GNUmakefile, makefile, or Makefile in the working directory.

$ make analysis
cc -c analy.c
cc -c stats.c
cc -o analysis analy.o stats.o

The following example also updates analysis but uses a makefile named analysis.mk in the working directory:

$ make -f analysis.mk analysis
'analysis' is up to date.

The next example lists the commands make would execute to bring the target-file named credit up-to-date. Because of the n option, make does not execute the commands.

$ make -n credit
cc -c -0 credit.c
cc -c -0 accounts.c
cc -c -0 terms.c
cc -o credit credit.c accounts.c terms.c

The next example uses the t option to update the modification time of the target-file named credit. After you use this option, make thinks that credit is up-to-date.

$ make -t credit
$ make credit
'credit' is up to date.

Next a simple makefile builds a utility named ff. Because the cc command needed to build ff is complex, using a makefile allows you to rebuild ff easily, without having to remember and retype the cc command.

$ cat Makefile
# Build the ff command from the fastfind.c source
SHELL=/bin/sh

ff:
cc -traditional -02 -g -DBIG=5120 -o ff fastfind.c myClib.a

$ make ff
cc -traditional -02 -g -DBIG=5120 -o ff fastfind.c myClib.a

Advanced example

The following example shows a much more sophisticated makefile that uses features not discussed in this section. Refer to the sources cited under "Documentation" on page 784 for information about these and other advanced features.

$ cat Makefile
###########################################################
## Build and maintain the buffer library
###########################################################

SHELL=/bin/sh
###########################################################
## Flags and libraries for compiling. The XLDLIBS are needed
#   whenever you build a program using the library. The CCFLAGS
#   give maximum optimization.
CC=gcc
CCFLAGS=-02 $(CFLAGS)
XLDLIBS= -lXaw3d -lXt -lXmu -lXext -lX11 -lm
BUFLIB=libbuf.a

###########################################################
## Miscellaneous
INCLUDES=buf.h
XINCLUDES=xtbuff_ad.h
OBJS=buff.o buf_print.o xtbuff.o

###########################################################
## Just a 'make' generates a help message
help:   Help
        @echo "You can make the following:"
        @echo " "
        @echo " libbuf.a       -- the buffer library"
        @echo " bufdisplay     -- display any-format buffer"
        @echo " buf2ppm        -- convert buffer to pixmap"
###########################################################
## The main target is the library
libbuf.a:  $(OBJS)
    -/bin/rm libbuf.a

    ar rv libbuf.a $(OBJS)
    ranlib libbuf.a
###########################################################
## Secondary targets -- utilities built from the library
bufdisplay: bufdisplay.c libbuf.a
    $(CC) $(CCFLAGS) bufdisplay.c -o bufdisplay $(BUFLIB) $(XLDLIBS)

buf2ppm: buf2ppm.c libbuf.a
    $(CC) $(CCFLAGS) buf2ppm.c -o buf2ppm $(BUFLIB)

###########################################################
## Build the individual object units
buff.o: $(INCLUDES) buff.c
    $(CC) -c $(CCFLAGS) buff.c

buf_print.o:$(INCLUDES) buf_print.c
    $(CC) -c $(CCFLAGS) buf_print.c

xtbuff.o: $(INCLUDES) $(XINCLUDES) xtbuff.c
    $(CC) -c $(CCFLAGS) xtbuff.c

The make utility can be used for tasks other than compiling code. As a final example, assume that you have a database that lists IP addresses and the corresponding hostnames in two columns and that the database dumps these values to a file named hosts.tab. You need to extract only the hostnames from this file and generate a Web page named hosts.html containing these names. The following makefile is a simple report writer:

$ cat makefile
#
SHELL=/bin/bash
#
hosts.html: hosts.tab
        @echo "<HTML><BODY>" > hosts.html
        @awk '{print $$2, "<br>"}' hosts.tab >> hosts.html
        @echo "</BODY></HTML>" >> hosts.html

man: Displays documentation for commands

man [options] [section] command

man k keyword

The man utility provides online documentation for Mac OS X commands. In addition to user commands, documentation is available for many other commands and details that relate to OS X. Because some OS X commands come from GNU, the GNU info utility (page 30) sometimes provides more complete information.

Each manual page begins with a one-line header. This header consists of a command name, the section of the manual in which the command is found, and a brief description of what the command does. These headers are stored in a database so that you can perform quick searches on keywords associated with each man page.

Arguments

The section argument tells man to limit its search to the specified section of the manual (see page 29 for a listing of manual sections). Without this argument, man searches the sections in numerical order and displays the first man page it finds. In the second form of the man command, the k option searches for the keyword in the database of man page headers; man displays a list of headers that contain the keyword. A man k command performs the same function as apropos (page 59).

Options

a | (all) Displays man pages for all sections of the manual. Without this option, man displays only the first page it finds. Use this option when you are not sure which section contains the information you are looking for.
k | keyword
 | (keyword) Displays manual page headers that contain the string keyword. You can scan this list for commands of interest. This option is equivalent to the apropos command (page 59).
K | keyword
 | (keyword search) Searches for keyword in all man pages. This option can take a long time to run.
M | path
 | (manpath) Searches the directories in path for man pages, where path is a colon-separated list of directories.
t | (troff) Formats the page for printing on a PostScript printer. The output goes to standard output.

Discussion

The manual pages are organized in sections, with each section pertaining to a separate aspect of the Mac OS X system. Section 1 contains user-callable commands and is most likely to be accessed by users who are not system administrators or programmers. Other sections of the manual describe system calls, library functions, and commands used by system administrators. See page 29 for a listing of the manual sections.

Pager

The man utility uses less to display manual pages that fill more than one screen. To change to another pager, set the environment variable PAGER to the pathname of the desired pager. For example, adding the following line to the ~/.bash_profile file allows a bash user to use more instead of less. Under OS X, less and more are both links to the same file, but they work slightly differently.

export PAGER=/usr/bin/more

MANPATH

You can tell man where to look for man pages by setting the environment variable MANPATH to a colon-separated list of directories. For example, bash users can add the following line to ~/.bash_profile to cause man to search the /usr/bin/man, /usr/share/man, and /usr/X11R6/man directories:

export MANPATH=/usr/bin/man:/usr/share/man:/usr/X11R6/man

As a privileged user, you can edit /usr/share/misc/man.conf to further configure man. Refer to the man man page for more information.

Notes

See page 28 for a discussion of man pages.

The argument to man is not always a command name. For example, the command man ascii lists the ASCII characters and their various representations; the command man k postscript lists man pages that pertain to PostScript.

The man pages are stored in an unformatted form. When you request a man page, it has to be formatted before being displayed.

Some utilities described in the manual pages have the same name as shell builtin commands. The behavior of the shell builtin may differ slightly from the behavior of the utility as described in the manual page. For information about shell builtins, see the man page for builtin or the man page for a specific shell.

References to man pages frequently use section numbers in parentheses; write(2) refers to the man page for write in section 2 of the manual (page 29).

Examples

The following example uses man to display the documentation for the command write, which sends messages to another user's terminal:

$ man write
WRITE(1)              BSD General Commands Manual                  WRITE(1)

NAME
write -- send a message to another user

SYNOPSIS
     write user [ttyname]

DESCRIPTION
The write utility allows you to communicate with other users, by copying
lines from your terminal to theirs.
...

The next example displays the man page for another commandthe man command itself, a good starting place for someone learning about the system:

$ man man
man(1)                                                            man(1)

NAME
       man - format and display the on-line manual pages

SYNOPSIS
 man [-acdfFhkKtwW] [--path] [-m system] [-p string] [-C config_file]
  [-M pathlist] [-P pager] [-S section_list] [section] name ...

DESCRIPTION
 man formats and displays the on-line manual pages. If you specify section,
 man only looks in that section of the manual. name is normally
 the name of the manual page, which is typically the name of a command,
 function, or file. However, if name contains a slash (/) then man
 interprets it as a file specification, so that you can do man ./foo.5
 or even man /cd/foo/bar.1.gz.
...

The next example demonstrates how you can use the man utility to find the man pages that pertain to a certain topic. In this case man k displays man page headers containing the string latex. The apropos utility (a shell script stored in /usr/bin/apropos) functions similarly to man k.

$ man -k latex
Pod::LaTeX(3pm)     - Convert Pod data to formatted Latex
pod2latex(1)        - convert pod documentation to latex format

The search for the keyword entered with the k option is not case sensitive. The 3pm on the first line indicates that the man page is from section 3 (Subroutines) of the UNIX System Manual and comes from the Perl Programmers Reference Guide (it is a Perl subroutine; see www.perl.org for more information on the Perl programming language).

mkdir: Creates a directory

mkdir [option] directory-list

The mkdir utility creates one or more directories.

Arguments

The directory-list is a list of one or more pathnames of directories that mkdir creates.

Options

m | mode
 | (mode) Sets the permission to mode. You can specify the mode absolutely by using an octal number (page 677) or symbolically (see Table VI-3 on page 676).
p | (parents) Creates any directories that do not exist in the path to the directory you wish to create.
v | (verbose) Displays the name of each directory created. This option is helpful when used with the p option.

Notes

You must have permission to write to and search (execute permission) the parent directory of the directory you are creating. The mkdir utility creates directories that contain the standard invisible entries (. and ..).

Examples

The following command creates the accounts directory as a subdirectory of the working directory and the prospective directory as a subdirectory of accounts:

$ mkdir -p accounts/prospective

Without changing working directories, the same user creates another subdirectory within the accounts directory:

$ mkdir accounts/existing

Next the user changes the working directory to the accounts directory and creates one more subdirectory:

$ cd accounts
$ mkdir closed

The last example shows the user creating another subdirectory. This time the m option removes all access permissions for the group and others:

$ mkdir -m go= accounts/past_due

mv: Renames or moves a file

mv [options] existing-file new-filename

mv [options] existing-file-list directory

mv [options] existing-directory new-directory

The mv utility, which renames or moves one or more files, has three formats. The first renames a single file with a new filename that you supply. The second renames one or more files so that they appear in a specified directory. The third renames a directory. The mv utility physically moves the file if it is not possible to rename it (that is, if you move the file from one filesystem [volume] to another).

Arguments

In the first form, the existing-file is a pathname that specifies the ordinary file that that mv is to rename. The new-filename is the new pathname of the file.

In the second form, the existing-file-list specifies the pathnames of the files that mv is to rename and the directory specifies the new parent directory for the files. The files you rename will have the same simple filenames as each of the files in the existing-file-list but new absolute pathnames.

The third form renames the existing-directory with the new-directory name. This form works only when the new-directory does not already exist.

Options

f | (force) Causes mv not to prompt you if a move would overwrite an existing file that you do not have write permission for. You must have write permission for the directory holding the target file.
i | (interactive) Prompts you for confirmation if mv would overwrite a file. If your response begins with a y or Y, mv overwrites the file; otherwise, mv does not move the file.
n | (no overwrite) Causes mv not to overwrite existing files, even if you have write permission for both the file and the directory holding it.
v | (verbose) Lists files as they are moved.

Notes

When mv copies a file to a different filesystem (volume), it first copies the existing-file to the new-file. It then deletes the existing-file. If the new-file already exists, mv may delete it before copying.

As with rm, you must have write and execute access permissions to the parent directory of the existing-file, but you do not need read or write access permission to the file itself. If the move would overwrite a file that you do not have write permission for, mv displays the file's access permissions and waits for a response. If you enter y or Y, mv overwrites the file; otherwise, it does not move the file. If you use the f option, mv does not prompt you for a response but simply overwrites the file.

Although earlier versions of mv could move only ordinary files between filesystems, mv can now move any type of file, including directories and device files.

Examples

The first command renames letter, a file in the working directory, as letter.1201:

$ mv letter letter.1201

The next command renames the file so that it appears, with the same simple filename, in the user's ~/archives directory:

$ mv letter.1201 ~/archives

The following command moves all files in the working directory whose names begin with memo so that they appear in the /p04/backup directory:

$ mv memo* /p04/backup

nice: Changes the priority of a command

nice [option] [command-line]

The nice utility reports the priority of the shell or alters the priority of a command. An ordinary user can decrease the priority of a command. Only Superuser can increase the priority of a command. The TC Shell includes a nice builtin that has a different syntax. Refer to "Notes" for more information.

Arguments

The command-line is the command line that nice is to execute at a different priority. Without any options or arguments, nice displays the priority of the shell running nice.

Options

Without an option, nice defaults to an adjustment of 10, lowering the priority of the command by 10typically from 0 to 10. As you raise the priority value, the command runs at a lower priority.

n | value
 | (nice) Changes the priority by the increment (or decrement) specified by value. The range of priorities is from 20 (the highest priority) to 20 (the lowest priority). A positive value lowers the priority, whereas a negative value raises the priority. Only Superuser can specify a negative value. When you specify a value outside this range, nice sets the priority to the limit of the range.

Notes

You can use renice (page 825) to change the priority of a running process.

Higher (more positive) priority values mean that the kernel schedules a job less often. Lower (more negative) values cause the job to be scheduled more often.

When Superuser schedules a job to run at the highest priority, this setting can affect the performance of the system for all other jobs, including the operating system itself. For this reason you should be careful when using nice with negative values.

tcsh

The TC Shell includes a nice builtin. Under tcsh, use the following syntax to change the priority at which command-line is run:

nice [±value] command line

The default priority is 4. You must include the plus sign for positive values.

Examples

The following command executes find in the background at the lowest possible priority. The ps l command displays the nice value of the command in the NI column:

# nice -n 20 find / -name '*.txt' -print > txtfiles.out &
[1] 2610
#ps -l
UID   PID PPID CPU PRI  NI  VSZ  RSS WCHAN  STAT  TT  TIME   COMMAND
501  9705 9701   0   31  0 27792  856 -      Ss    p1 0:00.15  -bash
501 10548 9705   0   12 20 27252  516 -      RN   p1 0:00.62 find  /
501   559  558   0   31  0 27792 1368 -       S    p2 0:00.02 -bash

The next command finds very large files and runs at a high priority (15):

# nice -n -15 find / -size +50000k

nidump: Displays contents of a NetInfo database

nidump database domain

The nidump (NetInfo dump) utility displays the contents of a NetInfo database.

Arguments

The database is the NetInfo database that nidump is to display. Table VI-18 lists the NetInfo databases. You must specify the domain, which is typically /, the default domain.

Table VI-18. NetInfo databases

aliases

group

protocols

bootparams

hosts

rpc

bootptab

networks

services

exports

passwd

fstab

printcap

Notes

The NetInfo databases hold the information that a traditional UNIX system stores in flat files with corresponding names in /etc. For example, the passwd and group databases specify the user and group account information stored in /etc/passwd and /etc/group on traditional UNIX systems.

The format of the passwd database under Mac OS X is different from that of the corresponding database under traditional UNIX. The OS X passwd database is kept in the 4.4BSD format, which adds three fields between the group ID (page 934) field and the user information or GECOS (page 934) field. The fields hold login class and account timestamp information that OS X does not use.

The nidump utility is useful for porting shell scripts that refer to traditional UNIX administration files. For example, if a script searches /etc/passwd for data, you can use nidump passwd / to list the records the script is looking for.

The nidump utility does not display encrypted passwords. Instead, it displays eight asterisks (********) in the password field or a single asterisk (*) if the account is disabled.

The lookupd daemon (page 455) provides consistent access to NetInfo databases, flat files, and other data sources. The nidump utility displays NetInfo databases only.

See page 441 for more information on NetInfo.

Examples

The first example displays the hosts database. This database is small because its only purpose is to allow the local system to look up names it may need before the network is configured. After that, lookupd passes these requests to DNS.

$ nidump hosts /
127.0.0.1       localhost
255.255.255.255 broadcasthost

The next example uses nidump to look at accounts that have the string root in their entries in the passwd database:

$ nidump passwd / | grep root
root:********:0:0::0:0:System Administrator:/private/var/root:/bin/sh
daemon:*:1:1::0:0:System Services:/var/root:/usr/bin/false

The final example displays the groups that have Alex as a member:

$ nidump group / | grep alex
appserverusr:*:79:appserver,alex
admin:*:80:root,alex
appserveradm:*:81:appserver,alex
alex:*:501:

nohup: Runs a command that keeps running after you log out

nohup command-line

The nohup utility executes a command line such that the command keeps running after you log out. In other words, nohup causes a process to ignore a SIGHUP signal. Depending on how the shell is configured, without nohup the shell may kill your background processes when you log out. The TC Shell includes a nohup builtin.

Arguments

The command-line is the command line that nohup is to execute.

Notes

Accepts the common options described on page 663.

If you do not redirect the output from a command that you execute using nohup, both standard output and standard error are sent to the file named nohup.out in the working directory. If you do not have write permission for the working directory, nohup sends output to ~/nohup.out.

Unlike the nohup utility, the TC Shell's nohup builtin does not send output to nohup.out. Background jobs started from tcsh continue to run after you log out.

Examples

The following command executes find in the background, using nohup:

$ nohup find / -name '*.txt' -print > txtfiles.out &
[1] 14235

od: Dumps the contents of a file

od [options] [file-list]

The od (octal dump) utility dumps the contents of a file. Such a dump is useful for viewing executable (object) files and text files that include embedded nonprinting characters. This utility takes its input from the files you specify on the command line or from standard input.

Arguments

The file-list specifies the pathnames of the files that od dumps. When you do not specify a file-list, od reads from standard input.

Options

A | base
 | (address radix) Specifies the base used to display the offsets shown for positions in the file. By default offsets are given in octal. Possible values for base are d (decimal), o (octal), x (hexadecimal), and n (no offsets displayed).
j | n
 | (jump) Skips n bytes before displaying data.
N | n
 | (maximum) Reads a maximum of n bytes and quits.
t | type [n]
 | Specifies the output format to use when displaying data from a file. You can repeat this option with different format types to see the file in several different formats. Table VI-19 lists the possible values for type.
 | By default od dumps a file as 2-byte octal numbers. You can specify the number of bytes od uses to compose each number by specifying a length indicator, n. You can specify a length indicator for all types except a and c. Table VI-21 lists the possible values of n.

Table VI-19. Output formats

type

Output format

a

Named character. Displays nonprinting control characters using their official ASCII names. For example, FORMFEED is displayed as ff.

c

ASCII character. Displays nonprinting control characters as backslash escape sequences (Table VI-20, next page) or three-digit octal numbers.

d

Signed decimal.

f

Floating point.

o

Octal (default).

u

Unsigned decimal.

x

Hexadecimal.

Table VI-20. Output format type c backslash escape sequences

Sequence

Meaning

\0

NULL

\a

BELL

\b

BACKSPACE

\n

NEWLINE

\r

RETURN

\t

HORIZONTAL TAB

\v

VERTICAL TAB

Table VI-21. Length indicators

n

Number of bytes to use

Integers (types d, o, u, and x)

C

(character)

Uses single characters for each decimal value

S

(short integer)

Uses 2 bytes

I

(integer)

Uses 4 bytes

L

(long)

Uses 4 bytes

Floating point (type f)

F

(float)

Uses 4 bytes

D

(double)

Uses 8 bytes

L

(long double)

Uses 16 bytes

Notes

To retain backward compatibility with older, non-POSIX versions of od, the od utility includes the options listed in Table VI-22 as shorthand versions of many of the preceding options.

Table VI-22. Shorthand format specifications

Shorthand

Equivalent specification

a

t a

b

t o1

c

t c

f

t fF

h

t x2

i

t dI

l

t dL

o

t o2

x

t x2

Examples

The file ac, which is used in the following examples, contains all of the ASCII characters. In the first example, the bytes in this file are displayed as named characters. The first column shows the offset of each byte from the start of the file. The offsets are given as octal values.

$ od -t a ac
0000000  nul  soh stx  etx  eot  enq ack bel  bs ht  nl  vt ff cr so  si
0000020  dle  dc1 dc2  dc3  dc4  nak syn etb can em sub esc fs gs rs  us
0000040   sp    !   "    #    $    %   &   '   (  )   *   +  ,  -  .   /
0000060    0    1   2    3    4    5   6   7   8  9   :   ;  <  =  >   ?
0000100    @    A   B    C    D    E   F   G   H  I   J   K  L  M  N   O
0000120    P    Q   R    S    T    U   V   W   X  Y   Z   [  \  ]  ^   _
0000140    `    a   b    c    d    e   f   g   h  i   j   k  l  m  n   o
0000160    p    q   r    s    t    u   v   w   x  y   z   {  |  }  ~ del
0000200   80   81  82   83   84   85  86  87  88 89  8a  8b 8c 8d 8e  8f
...
0000360   f0   f1  f2   f3   f4   f5  f6  f7  f8 f9  fa  fb fc fd fe  ff
0000400   nl
0000401

The next example displays the bytes as octal numbers, ASCII characters, or printing characters preceded by a backslash (refer to Table VI-20 on page 800):

$ od -t c ac
0000000  \0 001 002 003 004 005 006  \a  \b  \t  \n  \v  \f  \r 016 017
0000020 020 021 022 023 024 025 026 027 030 031 032 033 034 035 036 037
0000040       !   "   #   $   %   &   '   (   )   *   +   ,   -   .   /
0000060   0   1   2   3   4   5   6   7   8   9   :   ;   <   =   >   ?
0000100   @   A   B   C   D   E   F   G   H   I   J   K   L   M   N   O
0000120   P   Q   R   S   T   U   V   W   X   Y   Z   [   \   ]   ^   _
0000140   `   a   b   c   d   e   f   g   h   i   j   k   l   m   n   o
0000160   p   q   r   s   t   u   v   w   x   y   z   {   |   }   ~ 177
0000200 200 201 202 203 204 205 206 207 210 211 212 213 214 215 216 217
...
0000360 360 361 362 363 364 365 366 367 370 371 372 373 374 375 376 377
0000400  \n
0000401

The final example uses od to display the beginning of a file's resource fork (page 93). In some cases, information in the resource fork will give insight into the file's origins or contents. In this case, the file has been manipulated by the Graphic Converter application, a popular shareware image converter.

$ od -t c picture.jpg/..namedfork/rsrc | head
0000000      \0  \0 001  \0  \0 002   $   T  \0 002   #   T  \0  \0  \0 372
0000020      \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
*
0000400      \0  \0  \0 021 020   G   r   a   p   h   i   c   C   o   n   v
0000420       e   r   t   e   r  \0  \0  \0   ! 005     200  \0  \0  \0 003
0000440       5   .   2 026   (   f   o   r      G    r   a   p   h   i   c
0000460       C   o   n   v   e   r   t   e   r   )  \0  \0   _  \0   i   c
0000500       n   s  \0  \0   _  \0   i   c   h   #  \0  \0 002   H 377 377
0000520     377 377 376  \0 356 356 356 356 357  \0 335 335 335 335 336 200
0000540     340  \0  \0  \0 002   @ 240   B 022   " 002     322 001     201

The line with only an asterisk (*) indicates a series of identical lines.

open: Opens files, directories, and URLs

open [option] [file-list]

The open utility opens one or more files, directories, or URLs.

Arguments

The file-list specifies the pathnames of one or more files, directories, or URLs that open is to open.

Options

Without any options, open opens the files in file-list as though you had double-clicked each of the files' icons in the Finder.

a application
 | Opens file-list using application. Equivalent to dragging file-list to the application's icon in the Finder.
b bundle
 | Opens file-list using the application with bundle identifier bundle. A bundle identifier is a string, registered with the system, that identifies an application that can open files. For example, the bundle identifier com.apple.TextEdit specifies the TextEdit editor.
e | (edit) Opens file-list using the TextEdit application.
t | (text) Opens file-list using the default text editor (see "Discussion").
f | (file) Opens standard input as a file in the default text editor. Does not accept file-list.

Discussion

Opening a file brings up the application associated with that file. For example, opening a disk image file mounts it. The open command returns immediately, without waiting for the application to launch.

LaunchServices is a system framework that identifies applications that can open files. It maintains lists of available applications and user preferences about which application to use for each file type. LaunchServices also keeps track of the default text editor used by the t and f options.

Some options are not available under Mac OS X 10.3 and earlier.

Notes

When a file will be opened by a GUI application, you must run open from Terminal or another terminal emulator that is running under a GUI; otherwise the operation will fail.

Examples

The first example mounts the disk image file backups.dmg. The disk is mounted in /Volumes, using the name it was formatted with.

$ ls /Volumes
House Spare Tiger
$ open backups.dmg
$ ls /Volumes
Backups   House    Spare   Tiger

The next command opens the file picture.jpg. You must run this and the following example from a textual window within a GUI (e.g., Terminal). The application selected depends on the file attributes. If the file's type and creator code specify a particular application, open opens the file using that application. Otherwise, open uses the system's default program for handling .jpg files.

$ open picture.jpg

The final example opens the /usr/bin directory in the Finder. The /usr directory is normally hidden from the Finder because its invisible file attribute flag (page 95) is set. However, the open command can open any file you can access from the shell, even if it is not normally accessible from the Finder.

$ open /usr/bin

otool: Displays object, library, and executable files

otool options file-list

The otool utility displays information about, or part of, object, library, and executable files.

Arguments

The file-list specifies the pathnames of one or more files that otool is to display.

Options

You must use at least one of the L, M, t, or T options to specify which part of each file in file-list otool is to display.

L | (libraries) Displays the names and version numbers of the shared libraries an object file uses.
M | (module) Displays the module table of a shared library.
p | name
 | (print) Begins output at the symbol named name. Requires the t option and either the v or V option.
t | (text) Displays the TEXT section of an object file.
T | (table of contents) Displays the table of contents of a shared library.
v | (verbose) Displays data symbolically. When displaying code, this option causes otool to display the names of instructions instead of numeric codes.
V | (very verbose) Displays even more data symbolically. When displaying code, this option causes otool to display the names of called routines instead of their addresses.

Discussion

The otool utility displays information about the contents and dependencies of object files. You can use this information to help debug a program. For example, when setting up a chroot jail (page 461), otool can report which libraries are needed to run a given program.

Some options are useful only with some types of object modules. For example, the T option does not report anything for a typical executable file.

Notes

An otool L command performs a function similar to the ldd utility on systems using the ELF binary format.

Examples

The examples in this section use the compiled version of the following C program:

$ cat myname.c
#include <stdio.h>
int main(void) {
    printf("My name is Peter.\n");
    return 0;
}

In the first example, otool displays the libraries the program is linked with:

$ otool -L myname
myname:
    /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 92.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.0.0)

In some cases, a library used by a program will depend on other libraries. You can use otool -L on a library to see whether it uses other libraries:

$ otool -L /usr/lib/libmx.A.dylib /usr/lib/libSystem.B.dylib
/usr/lib/libmx.A.dylib:
    /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 92.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.0.0)
/usr/lib/libSystem.B.dylib:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.0.0)
    /usr/lib/system/libmathCommon.A.dylib (compatibility version 1.0.0, current ...

The next example disassembles the code for the main function. When compiling programs, the compiler sometimes modifies symbol names. The compiler gives functions, such as main, a leading underscore, so the symbol name is _main.

$ otool -Vt -p _main myname
myname:
(__TEXT,__text) section
_main:
00002ac0        mfspr   r0,lr
00002ac4        stmw    r30,0xfff8(r1)
00002ac8        stw     r0,0x8(r1)
00002acc        stwu    r1,0xffb0(r1)
00002ad0        or      r30,r1,r1
00002ad4        bcl     20,31,0x2ad8
00002ad8        mfspr   r31,lr
00002adc        addis   r2,r31,0x0
00002ae0        addi    r3,r2,0x4b8
00002ae4        bl      _printf$LDBLStub
00002ae8        li      r0,0x0
00002aec        or      r3,r0,r0
00002af0        lwz     r1,0x0(r1)
00002af4        lwz     r0,0x8(r1)
00002af8        mtspr   lr,r0
00002afc        lmw     r30,0xfff8(r1)
00002b00        blr
...

paste: Joins corresponding lines from files

paste [option] [file-list]

The paste utility reads lines from the file-list and joins corresponding lines in its output. By default, output lines are separated by a TAB character.

Arguments

The file-list specifies the pathnames of one or more ordinary files. When you omit the file-list, paste reads from standard input.

Options

d | dlist
 | (delimiter) The dlist is a list of characters that will be used to separate the output fields. If dlist contains a single character, paste uses that character instead of the default TAB character to separate fields. If dlist contains more than one character, the characters are used in turn to separate output fields and are then reused from the beginning of the list as necessary.
s | (serial) Processes one file at a time; pastes horizontally. See "Examples."

Notes

A common use of paste is to rearrange the columns of a table. A utility, such as cut, can place the desired columns in separate files, and then paste can join them in any order.

Examples

The following example uses the files fnames and acctinfo. These files can easily be created by using cut (page 699) and nidump (page 796). The paste command puts the full-name field first, followed by the remaining user account information. A TAB character separates the two output fields.

$ cat fnames
Jenny Chen
Alex Watson
Sam Adams
Helen Simpson

$ cat acctinfo
jenny:********:501:501::0:0:/Users/jenny:/bin/zsh
alex:********:502:502::0:0:/Users/alex:/bin/bash
sam:********:504:504::0:0:/Users/sam:/bin/tcsh
hls:********:505:505::0:0:/Users/hls:/bin/bash

$ paste fnames acctinfo
Jenny Chen         jenny:********:501:501::0:0:/Users/jenny:/bin/zsh
Alex Watson        alex:********:502:502::0:0:/Users/alex:/bin/bash
Sam Adams        sam:********:504:504::0:0:/Users/sam:/bin/tcsh
Helen Simpson      hls:********:505:505::0:0:/Users/hls:/bin/bash

The next examples use the files p1, p2, p3, and p4. The last example in this group uses the d option to give paste a list of characters to use to separate output fields.

$ cat p1
1
one
ONE
$ cat p2
2
two
TWO
extra
$ cat p3
3
three
THREE
$ cat p4
4
four
FOUR

$ paste p4 p3 p2 p1
4          3       2       1
four       three   two     one
FOUR       THREE   TWO     ONE
                   extra

$ paste -d "+-=" p3 p2 p1 p4
3+2-1=4
three+two-one=four
THREE+TWO-ONE=FOUR
+extra-=

The final example uses the s option to paste the files one at a time:

$ paste -s p1 p2 p3 p4
1          one     ONE
2          two     TWO         extra
3          three   THREE
4          four    FOUR

pax: Creates an archive, restores files from an archive, or copies a directory

pax [options] [pattern-list]

pax w [options] [source-files]

pax r [options] [pattern-list]

pax rw [options] destination-directory

The pax utility has four modes of operation: List mode displays the list of files in an archive, Create mode places multiple files into a single archive file, Extract mode restores files from an archive, and Copy mode makes a copy of a directory hierarchy. The archive files created by pax may be saved on disk, removable media, or a remote system.

List mode (no major options) reads an archive from standard input and displays the list of the files stored in the archive. Create mode (w for write) reads a list of ordinary or directory filenames from the command line or standard input and writes the resulting archive file to standard output. Extract mode (r for read) reads an archive from standard input and extracts files from that archive. Copy mode (rw) reads a list of ordinary or directory filenames from the command line or standard input and copies the files to an existing directory.

Arguments

In Create mode pax reads the names of files that will be included in the archive from source-files on the command line. If source-files is not present, pax reads the names from standard input.

In Extract mode pax extracts, and in List mode it displays the names of, all files found in the archive that it reads from standard input. You can choose to extract or display the names of files selectively by supplying a pattern-list. If the name of a file in the archive matches a pattern in the pattern-list, that file is extracted; otherwise, it is ignored. The pax patterns are similar to shell wildcards (page 133) except that they match slashes (/) and a leading period (.) in a filename. See the fnmatch man page for more information on pax patterns.

The pax utility does not create destination-directory in Copy mode. Instead, the destination-directory must exist before you give pax a Copy mode command.

Options

A major option specifies the mode in which pax operates: Create, Extract, or Copy. You can omit the other options.

Major Options

Three options determine the mode in which pax operates. You must include none or one of these options. Without a major option, pax runs in List mode.

r

(read) Puts pax in Extract mode, in which it reads the archive from standard input and extracts files. Without a pattern-list on the command line, pax exTRacts all files from the archive. With a pattern-list, pax exTRacts only those files with names that match one of the patterns in the pattern-list. The following example extracts from the root.pax archive file on an external drive only those files whose names end in .c:

$ pax -r \*.c < /Volumes/Backups/root.pax

The backslash prevents the shell from expanding the * before it passes the argument to pax.

w

(write) Puts pax in Create mode, in which it constructs an archive from the files named on the command line or standard input. These files may be ordinary or directory files, and each must appear on a separate line. The archive is written to standard output. The find utility frequently generates the filenames that pax uses. The following command builds an archive of the /Users directory and writes it to the archive file named Users.pax on /Volumes/Backups:

[View full width]
# find -d /Users -print | pax -w > /Volumes
  /Backups/Users.pax

The d option causes find to search for files in a depth-first manner, reducing the likelihood of permissions problems when you restore the files from the archive. See the discussion of this find option on page 696.

rw

(copy) Puts pax in Copy mode, in which it copies files named on the command line or standard input from one place on the system to another. Instead of constructing an archive file containing the files named on standard input, pax copies them into the destination-directory (the last argument on the pax command line). The effect is the same as if you had created an archive in Create mode and then extracted the files in Extract mode, except that using Copy mode avoids creating an archive.

The following example copies the working directory hierarchy into /Users/alex/code. The code directory must exist before you give this command. Make sure that ~alex/code is not a subdirectory of the working directory or you will perform a recursive copy.

$ pax -rw. ~alex/code

Other Options

The options listed in this section alter the behavior of pax.

c | (complement) Reverses the sense of the test done on pattern-list. Files are listed or put in the archive only if they do not match any of the patterns in pattern-list.
f | archive
 | (file) Uses archive as the name of the archive file. In List and Extract modes, reads from archive instead of standard input. In Create mode, writes to archive instead of standard output. You can use this option to access a device on another system on a network; see the --file option to tar (page 863) for more information.
H | (dereference) Copies the files that symbolic links given on the command line point to, not the symbolic links themselves. Does not copy the files that symbolic links found while descending a directory hierarchy point to.
l | (link) In Copy mode, when possible, makes hard links instead of copying files.
L | (dereference) Copies the files that symbolic links point to, not the symbolic links themselves. Includes the files that symbolic links found while descending a directory hierarchy point to.
P | (no dereference) Copies symbolic links, not the files that the links point to (default).
p | preserve-list
 | Preserves or discards the file attributes specified by preserve-list. The preserve-list is a string of one or more letters as shown in Table VI-23. By default pax preserves file access and modification times but does not preserve ownership or file permissions. This option works in Extract and Copy modes only.

Table VI-23. Preserve flags

Letter

Meaning

a

Discard access time

e

Preserve everything

m

Discard modification time

o

Preserve ownership

p

Preserve permissions

s

subcmd

Executes subcmd, a substitution command, on filenames while storing them in any of the modes. The subcmd has the following format:

s/search-string/replacement-string/[gp]

The pax utility replaces occurrences of the regular expression search-string with replacement-string. A trailing g indicates a global replacement; without it only the first instance of search-string in each filename is replaced. The p causes pax to display each substitution it makes. The subcmd is similar to vim's search and replace feature described on page 172, except that it lacks an address.

v

(verbose) In List mode, displays output similar to that produced by ls l. In other modes, displays a list of files being processed.

X

In Copy and Create modes, prevents pax from searching directories in filesystems other than the filesystem(s) that hold the files it was explicitly told to copy.

x

format

In Create mode, writes the archive in format format, as shown in Table VI-24. If you do not specify a format, pax writes a (POSIX) tar format file.

z

(gzip) In Create mode, compresses archives with gzip. In Extract mode, decompresses archives with gunzip.

Table VI-24. Archive formats

format

Description

cpio

The format specified for cpio archives in POSIX

sv4cpio

The format used for cpio archives under UNIX System V, release 4

tar

The historical Berkeley tar format (default)

ustar

The POSIX tar format

Discussion

The pax utility is a general replacement for tar, cpio, and other archive programs.

In Create and Copy modes, pax processes specified directories recursively. In List and Extract modes, if a pattern in pattern-list matches a directory name, pax lists or extracts the files from the named directory.

Notes

There is no native pax format. Instead, the pax utility can read and write archives in a number of formats. By default it creates POSIX tar format archives. See Table VI-24 for a list of formats supported by pax.

The pax utility determines the format of an archive file; it does not allow you to specify an archive format (x option) in List or Extract mode.

Under Mac OS X version 10.4 and later, pax archives extended attributes (page 93). Under version 10.3 and earlier, pax does not archive extended attributes. Under version 10.3 and earlier, you can use ditto (page 715) to copy files with their extended attributes.

Examples

In the first example, pax creates an archive named corres.0901.pax. This archive stores the contents of the corres directory.

$ pax -w corres > corres.0901.pax

The w option puts pax in Create mode, where the list of files to be put in an archive is supplied by command line arguments (corres in the preceding example) or from standard input. The pax utility sends the archive to standard output. The preceding example redirects that output to a file named corres.0901.pax.

Next without any options, pax displays a list of files in the archive it reads from standard input:

$ pax < corres.0901.pax
corres
corres/hls
corres/max
corres/max/0823
corres/max/0828
corres/max/0901
corres/memo1
corres/notes

The following command uses the f option to name the input file and performs the same function as the preceding command:

$ pax -f corres.0901.pax

When pax reads an archive file, as in the previous examples, it determines the format of the fileyou do not have to (nor are you allowed to) specify the format.

The next example attempts to create an archive of the /etc directory hierarchy in the default tar format. In this case pax is run by a user with root privileges because some of the files in this hierarchy cannot be read by ordinary users. Because pax does not follow (dereference) symbolic links by default, and because /etc is a symbolic link, pax copies the link and not the directory that the link points to. The first command creates the archive; the second command shows the link in the archive; and the third command uses the v option to show that it is a link:

# pax -w -f /tmp/etc.tar /etc
# pax -f /tmp/etc.tar
/etc
# pax -v -f /tmp/etc.tar
lrwxr-xr-x         1 root     admin         0 May 21 01:48 /etc => private/etc
pax: ustar    vol 1, 1 files, 10240 bytes read, 0 bytes written.

The L option follows (dereferences) links. In the next example, pax creates the desired archive of /etc:

# pax -wLf /tmp/etc.tar /etc
# pax -f /tmp/etc.tar
/etc
/etc/6to4.conf
/etc/AFP.conf
/etc/afpovertcp.cfg
/etc/aliases
/etc/aliases.db
/etc/amavisd.conf
/etc/amavisd.conf.personal
/etc/appletalk.cfg
...

The next example uses pax to create a backup of the memos directory, preserving ownership and file permissions. The destination directory must exist before you give pax a Copy mode command.

$ mkdir memos.0625
$ pax -rw -p e memos memos.0625
$ ls memos.0625
memos

The preceding example copies the memos directory into the destination directory. You can use pax to make a copy of a directory in the working directory without putting it in a subdirectory. In the next example, the s option causes pax to replace the name of the memos directory with the name . (the name of the working directory) as it writes files to the memos.0625 directory:

$ pax -rw -p e -s /memos/./ memos memos.0625

The final example uses find to build a list of files that begin with the string memo and that are located in the working directory hierarchy. The pax utility writes these files to an archive in the historical cpio format. The output from pax goes to standard output, which is sent through a pipe to bzip2, which in turn compresses it before writing it to the archive file.

$ find. -type f -name "memo*" | pax -w -x cpio | bzip2 > memos.cpio.bz2

plutil: Manipulates property list files

plutil [options] file-list

The plutil utility converts property list files (page 453) between formats and checks their syntax.

Arguments

The file-list specifies the pathnames of one or more files that plutil is to manipulate.

Options

convert format
 | Converts file-list to format, which must be either xml1 or binary1.
e extension
 | Gives output files a filename extension of extension.
help
 | Displays a help message.
lint
 | Checks the syntax of files (default).
o file
 | (output) Names the converted file file.
s | (silent) Does not display any messages for successfully converted files.

Discussion

The plutil utility converts files from the XML property list format to binary format, and vice versa. The plutil utility can read, but not write, the older plain-text property list format.

Notes

Refer to "Property List Files" on page 453 for a discussion of property list formats and more examples of the use of plutil.

The plutil utility accepts -- to mark the end of the options on the command line. For more information refer to "Common Options" on page 663

Examples

The following example checks the syntax of the file named java.plist.

$ plutil java.plist
java.plist: OK

The next example shows the output of plutil as it checks a damaged property list file:

$ plutil broken.plist
broken.plist:
XML parser error:
    Encountered unexpected element at line 2 (plist can only include one object)
Old-style plist parser error:
    Malformed data byte group at line 1; invalid hex

The next example converts the file StartupParameters.plist to binary format, overwriting the original file:

$ plutil -convert binary1 StartupParameters.plist

The final example converts the binary format property list file loginwindow.plist in the working directory to XML format and, because of the o option, puts the converted file in /tmp/lw.p:

$ plutil -convert xml1 -o /tmp/lw.p loginwindow.plist

pr: Paginates files for printing

pr [options] [file-list]

The pr utility breaks files into pages, usually in preparation for printing. Each page carries a header with the name of the file, date, time, and page number.

The pr utility takes its input from files you specify on the command line or from standard input. The output from pr goes to standard output and is frequently redirected by a pipe to a printer.

Arguments

The file-list specifies the pathnames of one or more text files that pr is to paginate. When you omit the file-list, pr reads from standard input.

Options

col
 | (columns) Displays the output in col columns with a default of one. This option may truncate lines and cannot be used with the m option.
d | (double-space) Double-spaces the output.
F | (formfeed) Uses a FORMFEED character to skip to the next page rather than filling the current page with NEWLINE characters.
h head
 | (header) Displays head at the top of each page instead of the filename. If head contains SPACEs, you must enclose it within quotation marks.
l lines
 | (lines) Sets the page length to lines lines. The default is 66 lines.
m | (merge) Displays all specified files simultaneously in multiple columns. This option cannot be used with the col option.
n [c[num]]
 | (number) Numbers the lines of output. The c is a character that pr appends to the number to separate it from the contents of the file (the default is a TAB). The num specifies the number of digits in each line number (the default is 5).
o spaces
 | (indent) Indents the output by spaces characters (specifies the left margin).
s[c]
 | (separate) Separates columns with the single character c (defaults to TAB when you omit c). By default pr uses SPACEs to align columns, padding them to a constant width.
t | (no trailer) Causes pr not to display its five-line page header and trailer. The header that pr normally displays includes the name of the file, the date, time, and page number. The trailer is five blank lines.
w num
 | (width) Sets the page width to num columns. This option is effective only with multicolumn output (the m or c option).
+firstpage
 | (start at) Output begins with the page numbered firstpage and runs through the last page of the document. This option begins with a plus sign, not a hyphen.

Notes

When you use the col option to display the output in multiple columns, pr displays the same number of lines in each column (with the possible exception of the last column).

Examples

The first command shows pr paginating a file named memo and sending its output through a pipe to lpr for printing:

$ pr memo | lpr

Now memo is sent to the printer again, this time with a special heading at the top of each page. The job is run in the background.

$ pr -h 'MEMO RE: BOOK' memo | lpr &
[1] 4904

Finally pr displays the memo file on the screen, without any header, starting with page 3:

$ pr -t +3 memo
...

ps: Displays process status

ps [options]

The ps utility displays status information about processes running on the local system.

Options

The ps utility accepts options without a leading hyphen for compatibility with the historical BSD ps utility. Without any options, ps reports on processes that you own. Unless you specify otherwise, ps sorts output by controlling terminal and then by PID number.

a | (all) Reports on all processes that are controlled by a terminal.
A | (all) Reports on all processes including those without a controlling terminal. Equivalent to ax.
e | (environment) Appends to the COMMAND field the environment of each process.
l | (long) Displays a lot of information about the processes. (See Table VI-26.)
u | (user oriented) For each process, displays the username of the user running the process, the time the process was started, the percentage of CPU and memory the process is using, and other information. The output is sorted by CPU usage.
U | username
 | (user) Reports on processes being run by username, a user on the local system.
w | (wide) Extends the display to 132 characters. A second w option causes ps to display complete lines, regardless of their length. Without this option, ps truncates output lines at the right side of the screen.
x | (extended) Reports on processes that you own, including those without a controlling terminal.

Discussion

Without any options, ps displays the statuses of all active processes that your terminal controls. Table VI-25 lists the heading and content of each of the five columns that ps displays when you call it without any options.

Table VI-25. Column headings I

Heading

Meaning

PID

The process identification number.

TT (terminal)

The name of the terminal that controls the process.

STAT

The current state of the process (see Table VI-26 for more information on this column).

TIME

The number of hours, minutes, and seconds the process has been running.

COMMAND

The command line the process was called with. The command is truncated to fit on one line. Use the w option to display more of the command line.

The columns that ps displays depend on your choice of options. Table VI-26 lists the headings and contents of the most common columns.

Table VI-26. Column headings II

Heading

Meaning

%CPU

The percentage of total CPU time that the process is using. Because of the way that OS X handles process accounting, this figure is approximate and the total of this column may exceed 100%.

%MEM (memory)

The percentage of RAM that the process is using. This value is approximate.

COMMAND

The command line the process was called with. The command is truncated to fit on one line. Use the w option to see more of the command line. This column is always displayed last on a line.

FLAGS

The flags associated with the process.

NI

nice value (priority).

PID

The process identification number.

PPID (parent PID)

The process identification number of the parent process.

PRI (priority)

The priority of the process.

RSS (resident set size)

The number of blocks of memory that the process is using.

STARTED

The time and date the process started.

STAT (status)

The status of the process as specified by one or more letters from the following list:

< | High priority
I | Idle (sleeping > ~20 seconds)
L | Pages locked in memory (real-time and custom I/O)
N | Low priority
R | Available for execution (in the run queue)
S | Sleeping (sleeping < ~20 seconds)
s | Session leader
T | Stopped
U | Sleeping and cannot be interrupted
W | Has no pages resident in RAM (swapped out)
X | Being traced or debugged
Z | Zombie process that is waiting for its child processes to terminate before it terminates

TIME

The number of minutes and seconds that the process has been running.

TT (terminal)

The name of the terminal controlling the process.

USER or UID

The username of the user who owns the process.

WCHAN (wait channel)

If the process is waiting for an event, the address of the kernel function that caused the process to wait. It is for processes that are not waiting or are sleeping.

Notes

Use top (page 874) to display process status information dynamically.

Examples

The first example shows ps, without any options, displaying the user's process that is running the shell (bash):

$ ps
     PID TT STAT    TIME COMMAND
   10607 p3 Ss   0:00.79 -bash

With the l (long) option, ps displays more information about the processes:

$ ps -l
UID   PID  PPID CPU PRI NI      VSZ  RSS WCHAN STAT TT        TIME COMMAND
502 10607 10603    0  31  0    27800  912 -      Ss  p3      0:00.79 -bash

The u option shows various types of information about the process, including how much of the local system CPU time and memory it is using:

$ ps -u
USER   PID %CPU %MEM      VSZ    RSS  TT  STAT STARTED     TIME COMMAND
alex 10607   0.0 -0.0    27800    908  p3  Ss   10:16PM   0:00.79 -bash

ps and kill

The next sequence of commands shows how to use ps to determine the PID number of a process running in the background and how to terminate that process by using the kill command. In this case it is not necessary to use ps because the shell displays the PID number of the background processes. The ps utility verifies the PID number.

The first command executes find in the background. The shell displays the job and PID numbers of the process, followed by a prompt.

$ find ~ -name memo -print > memo.out &
[1] 16292

Next ps confirms the PID number of the background task. If you did not already know this number, ps offers an easy way to discover it.

$ ps
     PID TT STAT     TIME COMMAND
   10607 p3 Ss    0:00.80 -bash
   16292 p3 U     0:01.12 find ~ -name memo -print

Finally kill (page 761) terminates the process:

$ kill 16292
$ RETURN
[1]+    Terminated               find ~ -name memo -print >memo.out
$

rcp: Copies one or more files to or from a remote system

rcp [options] source-file destination-file

rcp [options] source-file-list destination-directory

The rcp utility copies one or more ordinary files between two systems that can communicate over a network.

Security: rcp is not secure

The rcp utility uses host-based trust, which is not secure, to authorize files to be copied. Use scp (page 832) instead.

Arguments

The source-file, source-file-list, and destination-file specify pathnames of ordinary files, and the destination-directory specifies the pathname of a directory file. A pathname that does not contain a colon (:) identifies a file on the local system. A pathname of the form name@host:path specifies a file on the remote system named host. The path is relative to the home directory of the user name (unless path is an absolute pathname). When you omit the name@ portion of the destination, a relative pathname is relative to the home directory on the host of the user giving the rcp command.

Like cp, rcp has two modes of operation: The first copies one file to another file, and the second copies one or more files to a directory. The source-file[-list] is a list of the name(s) of the file(s) that rcp will copy; destination-file is the name that rcp assigns to the resulting copy of the file, or destination-directory is the name of the directory that rcp puts the copied files in. When rcp copies files to a destination-directory, the files maintain their original simple filenames.

Options

p | (preserve) Sets the modification time and file access permissions of each copy to match those of the source-file. When you do not use p, rcp uses the file-creation mask (umask; see page 883) on the remote system to modify the access permissions.
r | (recursive) When a file in the source-file-list is a directory, copies the contents of that directory and any subdirectories into the destination-directory. You can use this option only when the destination is a directory.

Notes

You must have an account on the remote system to copy files to or from it using rcp. The rcp utility does not prompt for a password but uses several alternative methods to verify that you have the authority to read or write files on the remote system.

One method requires that the name of the local system be specified in the /etc/hosts.equiv file on the remote system. If the name is there, rcp allows you to copy files if your usernames are the same on both systems and your account on the remote system has the permissions necessary to access files there.

Authorization can also be specified on a per-user basis. With this method, the remote user's home directory must contain a file named ~/. rhosts that lists trusted remote systems and users. Your local and remote usernames do not have to match in this case, but your local username must appear on the line in the remote ~/.rhosts file that starts with the name of the local system. See "Examples" for rlogin (page 826) for a sample .rhosts file.

If you use a wildcard (such as *) in a remote pathname, you must quote the wildcard character or pathname so that the wildcard is interpreted by the shell on the remote system and not by the local shell. As with cp, if the destination-file exists before you execute rcp, rcp overwrites the file.

For security reasons, the rcp service is not enabled by default on Mac OS X systems. Even without the service enabled, you can use rcp to connect to remote systems that provide an rcp service.

Examples

The first example copies the files with filenames ending in .c into the archives directory on the remote system named bravo. Because a username is not specified, rcp uses the local user's username on the remote system. Because the full pathname of the archives directory is not specified, rcp assumes that it is a subdirectory of the user's home directory on bravo. Each of the copied files retains its simple filename.

$ rcp *.c bravo:archives

The next example copies memo from the /Users/jenny directory on bravo to the working directory on the local system:

$ rcp bravo:/Users/jenny/memo .

Next rcp copies the files named memo.new and letter to Jenny's home directory on the remote system bravo. The absolute pathnames of the copied files on bravo are /Users/jenny/memo.new and /Users/jenny/letter:

$ rcp memo.new letter bravo:/Users/jenny

The final command copies all the files in Jenny's reports directory on bravo to the oldreports directory on the local system, preserving the original modification dates and file access permissions on the copies:

$ rcp -p 'bravo:reports/*' oldreports

renice: Changes the priority of a process

renice priority process-list

renice n increment process-list

The renice utility alters the priority of a running process. An ordinary user can decrease the priority of a process that he owns. Only Superuser can increase the priority of a process or alter the priority of another user's process.

Arguments

The process-list specifies the PID numbers of the processes that are to have their priorities altered. Each process has its priority set to priority or, in the second format, has its priority incremented by a value of increment (which can be negative).

Options

The options, which you can specify throughout process-list, change the interpretation of the arguments that follow them on the command line.

p | (process) Interprets the following arguments as process ID (PID) numbers.
u | (user) Interprets the following arguments as usernames or user ID numbers.

Notes

The range of priorities is from -20 (the highest priority) to +20 (the lowest priority). Higher (more positive) priority values mean that the kernel schedules a job less often. Lower (more negative) values cause the job to be scheduled more often.

When Superuser schedules a job to run at the highest priority, this change can affect the performance of the system for all other jobs, including the operating system itself. For this reason you should be careful when using renice with negative values.

See nice (page 794) if you want to start a process with a nondefault priority.

Examples

The first example decreases the priority of all tasks owned by Alex:

$ renice -n 5 -u alex

Following, root uses ps to check the priority of the process running find. The NI (nice) column shows a value of 19 and the administrator decides to increase the priority by 5:

#ps -l
UID   PID PPID CPU PRI NI      VSZ     RSS WCHAN  STAT  TT      TIME COMMAND
501  9705 9701   0  31  0     27792     856 -      Ss    p1    0:00.15 -bash
501 10548 9705   0  12 19     27252    516 -      RN    p1    0:00.62 find /
# renice -n -5 10548

rlogin: Logs in on a remote system

rlogin [option] remote-system

The rlogin utility establishes a login session on a remote system over a network.

Security: rlogin is not secure

The rlogin utility uses host-based trust, which is not secure, to authorize your login. Alternatively, it sends your password over the network as cleartext, which is not a secure practice. Use ssh (page 847) when it is available.

Arguments

The remote-system is the name of a system that the local system can reach over a network.

Options

l | username
 | (login) Logs you in on the remote system as the user specified by username.

Notes

If the file named /etc/hosts.equiv located on the remote system specifies the name of the local system, the remote system will not prompt you to enter your password. Systems that are listed in the /etc/hosts.equiv file are considered as secure as the local system.

An alternative way to specify a trusted relationship is on a per-user basis. Each user's home directory can contain a file named ~/.rhosts that holds a list of trusted remote systems and users. See "Examples" for a sample .rhosts file.

For security, the rlogin service is not enabled by default on Mac OS X systems. Even without the service enabled, you can use rlogin to connect to remote systems that provide an rlogin service.

Examples

The following example illustrates the use of rlogin. On the local system, Alex's username is alex; on the remote system bravo, his username is watson. The remote system prompts Alex to enter a password because he is logging in using a username different from the one he uses on the local system.

$ who am i
alex      tty06         Oct 14 13:26
$ rlogin -l watson bravo
Password:

~/.rhosts file

If the local system is named hurrah, the following .rhosts file on bravo allows the user alex to log in as the user watson without entering a password:

$ cat /Users/watson/.rhosts
hurrah alex

rm: Removes a file (deletes a link)

rm [options] file-list

The rm utility removes hard and/or symbolic links to one or more files. When you remove the last hard link to a file, the file is deleted.

Caution: Be careful when you use rm with wildcards

Because you can remove a large number of files with a single command, use rm cautiously, especially when you are working with ambiguous file references. If you have doubts about the effect of an rm command with an ambiguous file reference, first use echo with the same file reference and evaluate the list of files generated by the reference. Alternatively, you can use the i option.

Arguments

The file-list specifies the pathnames of one or more files that rm is to delete.

Options

f | (force) Without asking for your consent, removes files for which you do not have write access permission (you must have execute and write access permissions for the parent directory of the file). This option also suppresses informative output if a file does not exist.
i | (interactive) Asks before removing each file. If you use r with this option, rm also asks you before examining each directory.
r | (recursive) Deletes the contents of the specified directory, including all its subdirectories, and the directory itself. Use this option cautiously.
v | (verbose) Displays the name of each file as it is removed.

Notes

To delete a file, you must have execute and write access permissions for the parent directory of the file, but you do not need read or write access permission for the file itself. If you are running rm interactively (that is, if rm's standard input is coming from the keyboard) and you do not have write access permission for the file, rm displays your access permission and waits for you to respond. If your response starts with a y or Y, rm deletes the file; otherwise, it takes no action. If standard input is not coming from a keyboard, rm deletes the file without question.

Refer to page 103 for information on hard links and page 105 for information on symbolic links. Page 107 includes a discussion about removing links. Refer to the rmdir utility (page 829) if you need to remove an empty directory.

When you want to remove a file whose name begins with a hyphen, you must prevent rm from interpreting the filename as an option. One way to do so is to give the special option -- before the name of the file. This option tells rm that no more options follow: Any arguments that come after it are filenames, even if they look like options.

Security: rm does not remove a file securely

Using rm does not securely delete a fileit is possible to recover a file deleted with rm. See the example "Wiping a file" on page 704 for a way to delete files more securely.

Examples

The following commands delete files both in the working directory and in another directory:

$ rm memo
$ rm letter memo1 memo2
$ rm /Users/jenny/temp

The next example asks the user before removing each file in the working directory and its subdirectories:

$ rm -ir .

This command is useful for removing filenames that contain special characters, especially SPACEs, TABs, and NEWLINEs. (You should not create filenames containing these characters on purpose, but it may happen accidentally.)

rmdir: Removes a directory

rmdir directory-list

The rmdir utility deletes empty directories.

Arguments

The directory-list specifies the pathnames of empty directories that rmdir removes.

Options

p (parents) Removes a series of empty, nested directories, starting with the child.

Notes

Use the rm utility with the r option if you need to remove directories that are not empty, together with their contents.

Examples

The following command deletes the empty literature directory from the working directory:

$ rmdir literature

The next command removes the letters directory, using an absolute pathname:

$ rmdir /Users/jenny/letters

The final command removes the letters, march, and 05 directories, assuming the directories are empty except for other directories named in the path:

$ rmdir -p letters/march/05

rsh: Executes commands on a remote system

rsh [option] host [command-line]

The rsh utility runs command-line on host by starting a shell on the remote system. Without a command-line, rsh calls rlogin, which logs you in on the remote system.

Security: rsh is not secure

The rsh utility uses host-based trust, which is not secure, to authorize your login. Alternatively, it sends your password over the network as cleartext, which is not a secure practice. Use ssh (page 847) when it is available.

Arguments

The host is the name of the remote system. The rsh utility runs command-line on the remote system. You must quote special characters in command-line so that the local shell does not expand them prior to passing them to rsh.

Options

l | username
 | (login) Logs you in on the remote system as the user specified by username.

Notes

If the file named /etc/hosts.equiv located on the remote system specifies the name of the local system, the remote system will not prompt you to enter your password. Systems that are listed in the /etc/hosts.equiv file are considered as secure as the local system.

An alternative way to specify a trusted relationship is on a per-user basis. Each user's home directory can contain a file named ~/.rhosts that holds a list of trusted remote systems and users. See "Examples" under rlogin (page 826) for a sample .rhosts file.

For security reasons, the rsh service is not enabled by default on Mac OS X systems. Even without the service enabled, you can use rsh to connect to remote systems that provide an rsh service.

Examples

In the first example, Alex uses rsh to obtain a listing of the files in his home directory on bravo:

$ rsh bravo ls
cost_of_living
info
preferences
work

Next the output of the previous command is redirected into the file bravo.ls. Because the redirection character (>) is not escaped, it is interpreted by the local shell, and the file bravo.ls is created on the local system.

$ rsh bravo ls > bravo_ls
$ cat bravo_ls
cost_of_living
info
preferences
work

The next example quotes the redirection character (>) so that the file bravo.ls is created on the remote system (bravo), as shown by ls run on bravo:

$ rsh bravo ls ">" bravo.ls
$ rsh bravo ls
bravo.ls
cost_of_living
info
preferences
work

In the final example, rsh without command-line logs in on the remote system. Alex has used the l watson option to log in on bravo as watson. The /Users/watson/.rhosts file must be configured to allow Alex to log in on the account in this manner. See "Examples" under rlogin (page 826) for a sample .rhosts file.

$ rsh -l watson bravo
Last login: Sat Jul 30 16:13:53 from kudos
$ hostname
bravo
$ exit
rlogin: connection closed.

scp: Securely copies one or more files to or from a remote system

scp [[user@]from-host:]source-file [[user@]to-host:][destination-file]

The scp (secure copy) utility copies an ordinary or directory file from one system to another on a network. It uses ssh to transfer files and the same authentication mechanism as ssh; therefore it provides the same security as ssh. The scp utility asks you for a password when it is needed.

Arguments

The from-host is the name of the system you are copying files from; the to-host is the system you are copying files to. When you do not specify a host, scp assumes the local system. The user on either system defaults to the user on the local system who is giving the command; you can specify a different user with user@. The scp utility permits you to copy between two remote systems.

The source-file is the file you are copying, and the destination-file is the resulting copy. You can specify ordinary or directory files as relative or absolute pathnames. A relative pathname is relative to the specified or implicit user's home directory. When the source-file is a directory, you must use the r option to copy its contents. When the destination-file is a directory, each of the source files maintains its simple filename.

Options

E | (extended) Preserves extended attributes (page 93) when copying files. Requires both systems to be running Mac OS X 10.4 or later.
p | (preserve) Preserves the modification and access times as well as the permissions of the original file.
q | (quiet) Does not display the progress meter.
r | (recursive) Recursively copies a directory hierarchy.
v | (verbose) Displays debugging messages about the connection and transfer. This option is useful if things are not going as expected.

Notes

The scp utility is one of the OpenSSH suite of secure network connectivity tools. See "Notes" on page 848 for a discussion of OpenSSH security. Refer to "Message on initial connection to a server" on page 848 for information about a message you may get when using scp to connect to a remote system for the first time.

You can copy from or to the local system or between two remote systems. Make sure that you have read permission for the file you are copying and write permission for the directory you are copying it into.

You must quote a wildcard character (such as *) in a remote pathname so that it is interpreted by the shell on the remote system and not by the local shell.

As with cp, if the destination-file exists before you run scp, scp overwrites the file.

Under OS X version 10.3 and earlier, or when you do not use the E option under version 10.4 or later, scp does not preserve extended attributes (page 93).

Examples

The first example copies the files with filenames ending in .c from the working directory on the local system into the ~jenny/archives directory on bravo. The wildcard character is not quoted, so the local shell will expand it. Because archives is a relative pathname, scp assumes that it is a subdirectory of Jenny's home directory on bravo. Each of the copied files retains its simple filename.

$ scp *.c jenny@bravo:archives

Next Alex copies the directory structure under ~alex/memos on bravo to ~jenny/alex.memos.bravo on kudos. He must have the necessary permissions to write to Jenny's home directory on kudos.

$ scp -r bravo:memos jenny@kudos:alex.memos.bravo

Finally Alex copies the files with filenames ending in .c from Jenny's archives directory on bravo to the jenny.c.bravo directory in his working directory. The wildcard character is quoted to protect it from expansion by the local shell; it will be interpreted by the remote system, bravo.

$ scp -r 'jenny@bravo:archives/*.c' jenny.c.bravo

It is important to remember that whenever you copy multiple files or directories, the destinationeither local or remotemust exist and be a directory, not an ordinary file.

SetFile: Sets file attributes

SetFile [options] file-list

The SetFile utility sets file attributes (page 95), including the file's type and creator code, creation and last modification time, and attribute flags such as the invisible and locked flags.

Arguments

The file-list specifies the pathnames of one or more files that SetFile works on.

Options

The options for SetFile correspond to options for GetFileInfo (page 749).

a | flags
 | (attribute) Sets the attribute flags specified by flags. An uppercase letter for a flag sets that flag and a lowercase letter unsets the flag. The values of unspecified flags are not changed. See Table 4-3 on page 96 or the SetFile man page for a list of attribute flags.
c | creator
 | Sets the creator code to creator.
d | date
 | Sets the creation date to date. You must format the date as mm/dd/[yy]yy [hh:mm:[:ss] [AM | PM]]. If you do not specify AM or PM, SetFile assumes a 24-hour clock. You must enclose a date string that contains SPACEs within quotation marks.
m | date
 | (modification) Sets the modification date to date. The format of date is the same as that used with the d option.
t | type
 | Sets the type code to type.

Notes

The SetFile utility is available in /Developer/Tools when the Developer Tools package (page 478) is installed.

Minor differences distinguish the options to SetFile and the corresponding options to GetFileInfo. For example, you can specify multiple attribute flags with the a option to SetFile but only a single flag with GetFileInfo. Also, SetFile requires a SPACE between the a option and the list of flags; GetFileInfo does not allow a SPACE there.

Examples

The first example sets the type and creator codes of the file named arch to SIT5 and SIT!, respectively, indicating that it is a StuffIt archive. The GetFileInfo utility displays these codes. Because these utilities are not in the default PATH (page 285), you must either change your PATH variable or call them with absolute pathnames as shown:

$ /Developer/Tools/SetFile -t SIT5 -c SIT! arch
$ /Developer/Tools/GetFileInfo -c arch
"SIT!"
$ /Developer/Tools/GetFileInfo -t arch
"SIT5"

The next example marks the file named secret as invisible and locked. The file will not show up in the Finder, and most OS X applications will be unable to overwrite it.

$ /Developer/Tools/SetFile -a VL secret

The final example clears the invisible attribute flag from every file in the working directory:

$ /Developer/Tools/SetFile -a v *

sleep: Creates a process that sleeps for a specified interval

sleep time

The sleep utility causes the process executing it to go to sleep for the time specified.

Arguments

The time denotes the number of seconds that the process sleeps for. The time does not have to be an integer; you can specify a decimal fraction.

Examples

You can use sleep from the command line to execute a command after a period of time. The following example executes in the background a process that reminds you to make a phone call in 20 minutes (1,200 seconds):

$ (sleep 1200; echo "Remember to make call.") &
[1] 4660

You can also use sleep within a shell script to execute a command at regular intervals. The per shell script executes a program named heartbeat every 90 seconds:

$ cat per
#!/bin/bash
while true
do
    heartbeat
    sleep 90
done

If you execute a shell script such as per in the background, you can terminate it only by using kill.

The final shell script accepts the name of a file as an argument and waits for that file to be created. If the file does not exist, the script sleeps for 1 minute and 45 seconds before checking for the file again:

$ cat wait_for_file
#!/bin/bash

if [ $# != 1 ]; then
   echo "Usage: wait_for_file filename"
   exit 1
fi

while true
do
    if [ -f "$1" ]; then
        echo "$1 is here now"
        exit 0
    fi
    sleep 105
done

sort: Sorts and/or merges files

sort [options] [file-list]

The sort utility sorts and/or merges one or more text files.

Arguments

The file-list specifies the pathnames of one or more ordinary files that contain the text to be sorted. If the file-list is omitted, sort takes its input from standard input. Without the o option, sort sends its output to standard output. This utility sorts and merges files unless you use the m (merge only) or c (check only) option.

Options

When you do not specify an option, sort orders the file in the machine collating sequence (usually ASCII). Without a k option, sort orders a file based on full lines. Use k to specify sort fields within a line. You can follow a k option with additional options without a leading hyphen; see "Discussion" for more information. The sort utility accepts the standard help and version long options.

b | (blanks) Blanks (TAB and SPACE characters) normally mark the beginning of fields in the input file. Without this option, sort considers leading blanks to be part of the field they precede. This option ignores leading blanks within a field, so sort does not consider these characters in sort comparisons.
c | (check) Checks whether the file is properly sorted. The sort utility does not display anything if everything is in order. It displays a message if the file is not in sorted order and returns an exit status of 1.
d | (dictionary) Ignores all characters that are not alphanumeric or blanks. For example, sort does not consider punctuation with this option.
f | (fold) Considers all lowercase letters to be uppercase letters. Use this option when you are sorting a file that contains both uppercase and lowercase text.
help | Displays a help message.
i | (ignore) Ignores nonprinting characters.
k | start[,stop]
 | (key) Specifies a sort field within a line. Without this option, sort orders a file based on full lines. The sort field starts at the position on the line specified by start and ends at stop, or at the end of the line if stop is omitted. The start and stop pointers are in the format f[.c], where f is the field number and c is the optional character within the field. Numbering starts with 1. When c is omitted from start, it defaults to the first character in the field; when c is omitted from stop, it defaults to the last character in the field. See "Discussion" for further explanation of sort fields and "Examples" for illustrations of their use.
m | (merge) Assumes that multiple input files are all in sorted order and merges them without verifying that they are sorted.
n | (numeric) Sorts in arithmetic sequence; does not order lines or sort fields in the machine collating sequence. With this option, minus signs and decimal points take on their arithmetic meanings.
o | filename
 | (output) Sends the output to filename instead of standard output; filename can be the same as one of the names in the file-list.
r | (reverse) Reverses the sense of the sort (for example, z precedes a).
t | x
 | (separator) Specifies x as the field separator. See "Discussion" for more information on field separators.
version | Displays a version message.
u | (unique) Outputs repeated lines only once. When you use this option with c, sort displays a message if the same line appears more than once in the input file, even if the file is in sorted order.

Discussion

Without any options, sort bases its ordering on full lines.

In the following description, a field is a sequence of characters in a line of input. Without the t option, fields are bounded by the empty string preceding a group of one or more blanks (TAB and SPACE characters). You cannot see the empty string that delimits the fields; it is an imaginary point between two fields. Fields are also bounded by the beginning and end of the line. The line shown in (Figure VI-1) holds the fields Toni, SPACE Barnett, and SPACESPACESPACESPACE 55020. These fields are used to define sort fields. Sometimes fields and sort fields are the same.

Figure VI-1. Fields and sort fields

Sort field

A sort field is a sequence of characters that sort uses to put lines in order. It can contain all or part of one or more fields (Figure VI-1).

The k option specifies pairs of pointers that define subsections of each line (sort fields) for comparison. See the k option for details.

Leading blanks

The b option causes sort to ignore leading blanks in a sort field. If you do not use this option, sort considers each leading blank to be a character in the sort field and includes it in the sort comparison.

Options

You can specify options that pertain only to a given sort field by immediately following the stop pointer (or the start pointer if there is no stop pointer) with one of the options b, d, f, i, n, or r. In this case you must not precede the option with a hyphen.

Multiple sort fields

When you specify more than one sort field, sort examines the sort fields in the order that they appear on the command line. If the first sort field of two lines is the same, sort examines the second sort field. If these are again the same, sort looks at the third field. This process continues for all the sort fields you specify. If all the sort fields are the same, sort examines the entire line.

Examples

The examples in this section demonstrate some of the features and uses of the sort utility. The examples assume that the list file shown here is in the working directory:

$ cat list
Tom Winstrom           94201
Janet Dempsey          94111
Alice MacLeod          94114
David Mack             94114
Toni Barnett           95020
Jack Cooper            94072
Richard MacDonald      95510

This file contains a list of names and ZIP codes. Each line of the file contains three fields: the first name field, the last name field, and the ZIP code field. For the examples to work, make sure the blanks in the file are SPACEs, not TABs.

The first example demonstrates sort without any optionsthe only argument is the name of the input file. In this case sort orders the file on a line-by-line basis. If the first characters on two lines are the same, sort looks at the second characters to determine the proper order. If the second characters are the same, sort looks at the third characters. This process continues until sort finds a character that differs between the lines. If the lines are identical, it does not matter which one sort puts first. In this example, sort needs to examine only the first three characters (at most) of each line. The sort utility displays a list that is in alphabetical order by first name.

$ sort list
Alice MacLeod          94114
David Mack             94114
Jack Cooper            94072
Janet Dempsey          94111
Richard MacDonald      95510
Tom Winstrom           94201
Toni Barnett           95020

You can instruct sort to skip any number of fields and characters on a line before beginning its comparison. Blanks normally mark the beginning of a field. The next example sorts the same list by last name, the second field. The k 2 argument instructs sort to begin its comparison with the second field. Because there is no second pointer, the sort field extends to the end of the line. Now the list is almost in last-name order, but there is a problem with Mac.

$ sort -k 2 list
Toni Barnett           95020
Jack Cooper            94072
Janet Dempsey          94111
Richard MacDonald      95510
Alice MacLeod          94114
David Mack             94114
Tom Winstrom           94201

In the preceding example, MacLeod comes before Mack. After finding that the sort fields of these two lines were the same through the third letter (Mac), sort put L before k because it arranges lines based on ASCII character codes, in which uppercase letters come before lowercase ones.

The f option makes sort treat uppercase and lowercase letters as equals and fixes the problem with MacLeod and Mack:

$ sort -f -k 2 list
Toni Barnett           95020
Jack Cooper            94072
Janet Dempsey          94111
Richard MacDonald      95510
David Mack             94114
Alice MacLeod          94114
Tom Winstrom           94201

The next example attempts to sort list on the third field, the ZIP code. In this case sort does not put the numbers in order but rather puts the shortest name first in the sorted list and the longest name last. The k 3 argument instructs sort to begin its comparison with the third field. A field starts with a blank and includes subsequent blanks. In the case of the list file, the blanks are SPACEs. The ASCII value of a SPACE character is less than that of any other printable character, so sort puts the ZIP code that is preceded by the most SPACEs first and the ZIP code that is preceded by the fewest SPACEs last.

$ sort -k 3 list
David Mack             94114
Jack Cooper            94072
Tom Winstrom           94201
Toni Barnett           95020
Janet Dempsey          94111
Alice MacLeod          94114
Richard MacDonald      95510

The b option causes sort to ignore leading SPACEs within a field. With this option, the ZIP codes come out in the proper order. When sort determines that MacLeod and Mack have the same ZIP codes, it compares the entire lines, putting Alice MacLeod before David Mack (because A comes before D).

$ sort -b -k 3 list
Jack Cooper            94072
Janet Dempsey          94111
Alice MacLeod          94114
David Mack             94114
Tom Winstrom           94201
Toni Barnett           95020
Richard MacDonald      95510

To sort alphabetically by last name when the ZIP codes are the same, sort needs to make a second pass that sorts on the last name field. The next example shows how to make this second pass by specifying a second sort field and uses the f option to keep the Mack/MacLeod problem from cropping up again:

$ sort -b -f -k 3 -k 2 list
Jack Cooper             94072
Janet Dempsey           94111
David Mack              94114
Alice MacLeod           94114
Tom Winstrom            94201
Toni Barnett            95020
Richard MacDonald       95510

The next example shows a sort command that skips not only fields but also characters. The k 3.4 option causes sort to start its comparison with the fourth character of the third field. Because the command does not define an end to the sort field, it defaults to the end of the line. The sort field is the last two digits in the ZIP code.

$ sort -fb -k 3.4 list
Tom Winstrom 94201
Richard MacDonald 95510
Janet Dempsey 94111
Alice MacLeod 94114
David Mack 94114
Toni Barnett 95020
Jack Cooper 94072

The problem of how to sort by last name within the last two digits of the ZIP code is solved by making a second pass covering the last-name field. The f option following the k 2 affects the second pass, which orders by last name only.

$ sort -b -k 3.4 -k 2f list
Tom Winstrom            94201
Richard MacDonald       95510
Janet Dempsey           94111
David Mack              94114
Alice MacLeod           94114
Toni Barnett            95020
Jack Cooper             94072

The next set of examples uses the cars data file. Each column in this file is separated from the next by SPACEs, not TABs. From left to right, the columns in the file contain each car's make, model, year of manufacture, mileage, and price:

$ cat cars
plym    fury    1970    73    2500
chevy   malibu  1999    60    3000
ford    mustang 1965    45    10000
volvo   s80     1998    102   9850
ford    thundbd 2003    15    10500
chevy   malibu  2000    50    3500
bmw     325i    1985    115   450
honda   accord  2001    30    6000
ford    taurus  2004    10    17000
toyota  rav4    2002    180   750
chevy   impala  1985    85    1550
ford    explor  2003    25    9500

Without any options, sort displays a sorted copy of the file:

$ sort cars
bmw      325i    1985    115    450
chevy    impala  1985    85     1550
chevy    malibu  1999    60     3000
chevy    malibu  2000    50     3500
ford     explor  2003    25     9500
ford     mustang 1965    45     10000
ford     taurus  2004    10     17000
ford     thundbd 2003    15     10500
honda    accord  2001    30     6000
plym     fury    1970    73     2500
toyota   rav4    2002    180    750
volvo    s80     1998    102    9850

The objective of the next example is to sort by manufacturer and by price within manufacturer. Unless you specify otherwise, a sort field extends to the end of the line. The k 1 sort field specifier sorts from the beginning of the line. The command line instructs sort to sort on the entire line and then make a second pass, sorting on the fifth field all lines whose first-pass sort fields were the same (k 5):

$ sort -k 1 -k 5 cars
bmw      325i      1985    115    450
chevy    impala    1985    85     1550
chevy    malibu    1999    60     3000
chevy    malibu    2000    50     3500
ford     explor    2003    25     9500
ford     mustang   1965    45     10000
ford     taurus    2004    10     17000
ford     thundbd   2003    15     10500
honda    accord    2001    30     6000
plym     fury      1970    73     2500
toyota   rav4      2002    180    750
volvo    s80       1998    102    9850

Because no two lines are the same, sort makes only one pass, sorting on each entire line. (If two lines differed only in the fifth field, they would be sorted properly on the first pass anyway, so the second pass would be unnecessary.) Look at the lines containing taurus and thundbd. They are sorted by the second field rather than the fifth field, demonstrating that sort never made a second pass and so never sorted on the fifth field.

The next example forces the first-pass sort to stop at the end of the first field. The k 1,1 option specifies a start pointer of the first character of the first field and a stop pointer of the last character of the first field. When you do not specify a character within a start pointer, it defaults to the first character; when you do not specify a character within a stop pointer, it defaults to the last character. Now taurus and thundbd are properly sorted by price. But look at explor: It is less expensive than the other Fords, but sort has it positioned as the most expensive. The sort utility put the list in ASCII collating sequence order, not in numeric order: Thus 9500 comes after 10000 because 9 comes after 1.

$ sort -k 1,1 -k 5 cars
bmw      325i       1985    115    450
chevy    impala     1985    85     1550
chevy    malibu     1999    60     3000
chevy    malibu     2000    50     3500
ford     mustang    1965    45     10000
ford     thundbd    2003    15     10500
ford     taurus     2004    10     17000
ford     explor     2003    25     9500
honda    accord     2001    30     6000
plym     fury       1970    73     2500
toyota   rav4       2002    180    750
volvo    s80        1998    102    9850

The n (numeric) option on the second pass puts the list in the proper order:

$ sort -k 1,1 -k 5n cars
bmw      325i       1985    115    450
chevy    impala     1985    85     1550
chevy    malibu     1999    60     3000
chevy    malibu     2000    50     3500
ford     explor     2003    25     9500
ford     mustang    1965    45     10000
ford     thundbd    2003    15     10500
ford     taurus     2004    10     17000
honda    accord     2001    30     6000
plym     fury       1970    73     2500
toyota   rav4       2002    180    750
volvo    s80        1998    102    9850

The next example again demonstrates that, unless you instruct it otherwise, sort orders a file starting with the field you specify and continuing to the end of the line. It does not make a second pass unless two of the first sort fields are the same. Because there is no stop pointer on the first sort field specifier, the sort field for the first pass includes the third field through the end of the line. Although this example sorts the cars by years, it does not sort them by model within manufacturer within years (ford thndbd comes before ford explor, when these lines should be reversed).

$ sort -k 3 -k 1 cars
ford     mustang  1965    45     10000
plym     fury     1970    73     2500
bmw      325i     1985    115    450
chevy    impala   1985    85     1550
volvo    s80      1998    102    9850
chevy    malibu   1999    60     3000
chevy    malibu   2000    50     3500
honda    accord   2001    30     6000
toyota   rav4     2002    180    750
ford     thundbd  2003    15     10500
ford     explor   2003    25     9500
ford     taurus   2004    10     17000

Specifying an end to the sort field for the first pass allows sort to perform its secondary sort correctly:

$ sort -k 3,3 -k 1 cars
ford    mustang  1965     45     10000
plym    fury     1970     73     2500
bmw     325i     1985     115    450
chevy   impala   1985     85     1550
volvo   s80      1998     102    9850
chevy   malibu   1999     60     3000
chevy   malibu   2000     50     3500
honda   accord   2001     30     6000
toyota  rav4     2002     180    750
ford    explor   2003     25     9500
ford    thundbd  2003     15     10500
ford    taurus   2004     10     17000

The next examples demonstrate several important sorting techniques: putting a list in alphabetical order, merging uppercase and lowercase entries, and eliminating duplicates. The unsorted list follows:

$ cat short
Pear
Pear
apple
pear
Apple

Following is a plain sort:

$ sort short
Apple
Pear
Pear
apple
pear

The following folded sort is a good start, but it does not eliminate duplicates:

$ sort -f short
Apple
apple
Pear
Pear
pear

The u (unique) option eliminates duplicates, but without the f the uppercase entries come first:

$ sort -u short
Apple
Pear
apple
pear

When you attempt to use both u and f, some of the entries get lost:

$ sort -uf short
apple
Pear

Two passes is the answer. Both passes are unique sorts, and the first folds lowercase letters onto uppercase ones:

$ sort -u -k 1f -k 1 short
Apple
apple
Pear
pear

split: Divides a file into sections

split [options] [filename [prefix]]

The split utility breaks its input into 1,000-line sections named xaa, xab, xac, and so on. The last section may be shorter than the others. Options can change the sizes of the sections and lengths of the names.

Arguments

The filename is the pathname of the file that split processes. If you do not specify an argument or if you specify a hyphen () instead of a filename, split reads from standard input. The prefix is one or more characters that split uses to prefix the names of the files it creates. The default prefix is x.

Options

a | len
 | (suffix length) Specifies that the filename suffix is len characters long (the default is 2).
b | n[u]
 | (bytes) Breaks the input into files that are n bytes long. The u is an optional unit of measure (k for kilobytes or m for megabytes). If you include a unit of measure, split counts by this unit instead of by bytes.
l | num
 | (lines) Breaks the input into files that are num lines long (the default is 1,000).

Discussion

By default split names the first file it creates xaa. The x is the default prefix. You can change this prefix with the prefix argument. You can change the number of characters in each filename following the prefix with the a option.

Examples

By default split breaks a file into 1,000-line sections with the names xaa, xab, xac, and so on. The wc utility with the l option shows the number of lines in each file. The last file, xar, is smaller than the rest.

$ split bigfile
$ wc -l xa*
  1000 xaa
  1000 xab
  1000 xac
 ...
  1000 xap
  1000 xaq
   103 xar
 17103 total

The next example uses the prefix argument to specify a filename prefix of SEC and uses a to change the number of letters in the filename suffix to 3:

$ split -a 3 /etc/services SEC
$ ls
SECaaa  SECaac  SECaae  SECaag  SECaai  SECaak
SECaab  SECaad  SECaaf  SECaah  SECaaj  SECaal

ssh: Securely executes commands on a remote system

ssh [option] [user@]host [command-line]

The ssh (secure shell) utility runs command-line on host by starting a shell on the remote system or logs you in on host. The ssh utility, which can replace rsh and rlogin, provides secure, encrypted communication between two systems on an insecure network.

Arguments

The host is the system that you want to log in or run a command on. Unless you have one of several kinds of authentication established, ssh prompts you for a username and password for the remote system. When ssh is able to log in automatically, it logs in as the user running the ssh command or as user if user@ appears on the ssh command line.

The command-line runs on the remote system. Without command-line, ssh logs you in on the remote system. You must quote special characters in command-line if you do not want the local shell to expand them.

Options

f | (not foreground) Sends ssh to the background after asking for a password and before executing command-line. This option is useful when you want to run the command-line in the background but must supply a password. Its use implies n.
l | user
 | (login) Attempts to log in as user. This option is equivalent to using user@ on the command line.
n | (null) Redirects standard input to the ssh utility to come from /dev/null (page 444). See f.
p | port
 | (port) Connects to port port on the remote host.
q | (quiet) Suppresses warning and diagnostic messages.
t | (tty) Allocates a pseudo-tty to the ssh process on the remote system. Without this option, when you run a command on a remote system, ssh does not allocate a tty (terminal) to the process. Instead, ssh attaches standard input and standard output of the remote process to the ssh sessionthat is normally, but not always, what you want. This option forces ssh to allocate a tty on the remote system so that programs that require a tty will work.
v | (verbose) Displays debugging messages about the connection and transfer. This option is useful if things are not going as expected.
x | (X11) Turns off X11 forwarding.
X | (X11) Turns on X11 forwarding. You may not need this optionX11 forwarding may be turned on in a configuration file.

Notes

OpenSSH security

Using public-key encryption, OpenSSH provides two levels of authentication: server and client/user. First, the client (ssh or scp) verifies that it is connected to the correct server and OpenSSH encrypts communication between the client and the server. Second, once a secure, encrypted connection has been established, OpenSSH confirms that the user is authorized to log in on or copy files from and to the server. Once the system and user have been verified, OpenSSH allows different services to pass through the connection. These services include interactive shell sessions (ssh), remote command execution (ssh and scp), X11 client/server connections, and TCP/IP port tunneling.

Message on initial connection to a server

When you connect to an OpenSSH server for the first time, the OpenSSH client prompts you to confirm that you are connected to the correct system. This check can help prevent a person-in-the-middle attack.

The authenticity of host 'grape (192.168.0.3)' can't be established.
RSA key fingerprint is c9:03:c1:9d:c2:91:55:50:e8:19:2b:f4:36:ef:73:78.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'grape,192.168.0.3' (RSA) to the list of
known hosts.

Before you respond to the preceding query, verify that you are logging in on the correct system and not on an imposter. If you are not sure, a telephone call to someone who logs in on that system locally can verify that you are on the intended system. When you answer yes (you must spell it out), the client appends the server's public host key to the user's ~/.ssh/known_hosts file on the local system, creating the ~/.ssh directory if necessary. So that it can keep track of which line in known_hosts applies to which server, OpenSSH prepends the name of the server and the server's IP address to the line. Subsequently, when you use OpenSSH to connect to that server, the client verifies that it is connected to the correct server by comparing this key to the one supplied by the server.

Examples

In the first example, Alex uses ssh to display a list of the files in his home directory on kudos:

$ ssh kudos ls
alex@kudos's password:
Work
code
graphs
reports

Next the output of the previous command is redirected to the file kudos_ls. Because the redirection character (>) is not escaped, it is interpreted by the local shell, and the file kudos_ls is created on the local system.

$ ssh kudos ls > kudos_ls
alex@kudos's password:
$ cat kudos_ls
Work
code
graphs
reports

The next example quotes the entire command that will run on the remote system. As a result the local shell does not interpret the redirection character (>) but rather passes it to the remote shell. The file kudos.ls is created on the remote system (kudos), as shown by ls run on kudos:

$ ssh kudos "ls > kudos.ls"
alex@kudos's password:
$ ssh kudos ls
alex@kudos's password:
Work
code
graphs
kudos.ls
reports

The next command does not quote the pipe symbol (|). As a result the pipe is interpreted by the local shell, which sends the output of the remote ls to standard input of less on the local system:

$ ssh kudos ls | less

Next ssh executes a series of commands, connected with pipes, on a remote system. The commands are enclosed within single quotation marks so that the local shell does not interpret the pipe symbols and all the commands are run on the remote system.

$ ssh kudos 'ps -ef | grep nmbd | grep -v grep | cut -c10-15 |xargs kill -1'

The output of ps is piped through grep, which passes all lines containing the string nmbd to another invocation of grep. The second grep passes all lines not containing the string grep to cut (page 699). The cut utility extracts the process ID numbers and passes them to xargs (page 892), which kills the listed processes with a HUP signal (kill 1).

In the following example, ssh without command-line logs in on the remote system. Here Alex has used watson@kudos to log in on kudos as watson:

$ ssh watson@kudos
watson@kudos's password:
Last login: Sat Sep 17 06:51:59 from bravo
$ hostname
kudos
$ exit

Alex now decides to change the password for his watson login on kudos:

$ ssh watson@kudos passwd
Password:
Old password:por

Alex stops as soon as he sees passwd (running on kudos) displaying his password: He knows that something is wrong. For the passwd to work, it must run with a tty (terminal) so that it can turn off character echo (stty echo) and thus not display a password as the user enters it. The t option solves this problem by associating a pseudo-tty with the process running passwd on the remote system:

$ ssh -t watson@kudos passwd
watson@kudos's password:
Changing password for watson
Old password:
New password:
Retype new password:
Connection to kudos closed.
$

The t option is also useful when you are running a program that uses a character-based/pseudographical interface.

The final example uses tar (page 862) to create an archive file of the contents of the working directory hierarchy. The f option causes tar to send its output to standard output. A pipe sends the output of tar running on the local system, via ssh, to dd (page 703) running on the remote system.

$ cat buwd
#! /bin/bash
# back up the working directory to the user's
# home directory on the remote system specified
# by $machine

# remote system:
machine=speedy

dir=$(basename $(pwd))
filename=$$.$dir.tar

echo Backing up $(pwd) to your home directory on $machine
tar -cf - . | ssh $machine "dd obs=256k of=$filename"
echo done. Name of file on $machine is $filename

stat: Displays information about files

stat [options] [file-list]

The stat utility displays information about files.

Arguments

The file-list specifies the pathnames of one or more files that stat displays information about. Without a file-list, stat displays information about standard input.

Options

Without any options, stat displays all of the available information about each file it processes.

F | (file type) Displays a slash (/) after each directory, an asterisk (*) after each executable file, an at sign (@) after a symbolic link, an equal sign (=) after a socket, and a pipe (|) after a FIFO.
L | (dereference) Displays information about the file a link points to, not the link itself.
q | (quiet) Suppresses error messages.
l | (long) Uses the same format as ls l (page 87).
s | (shell) Displays information in a format that can be used to initialize shell variables. See the "Examples" section.
x | (Linux) Displays a more verbose format that is compatible with the version of stat found on some Linux distributions.
f | format
 | Formats output using format. The format string is similar to that used by printf. See the "Examples" section and the stat man page for more information.

Discussion

The stat utility is a front end for the stat() system call (give the command man 2 stat to see the man page). The stat utility displays the permissions and timestamps of a file, but not its file attributes (page 95). Use GetFileInfo (page 749) to display file attributes.

Notes

The stat utility was introduced in Mac OS X 10.4.

Examples

By default stat does not label the information it displays. The first example displays information about the root filesystem (startup volume):

$ stat /
234881026 2 drwxrwxr-t 41 root admin 0 1496 "Jul 19 23:41:32 2005" "Jul 17 23:21:05 2005"
"Jul 17 23:21:05 2005" 4096 0 0 /

The first value in the preceding output is the device number. The x option formats and labels the information, with the device number displayed as a pair of numbers separated by a comma:

$ stat -x /
  File: "/"
  Size: 1496         FileType: Directory
  Mode: (0775/drwxrwxr-t)         Uid: (    0/    root)  Gid: (   80/   admin)
Device: 14,2   Inode: 2    Links: 41
Access: Tue Jul 19 23:41:32 2005
Modify: Sun Jul 17 23:21:05 2005
Change: Sun Jul 17 23:21:05 2005

The s option displays information as a set of shell variable assignments. The names of the variables correspond to the members of the structure returned by the stat() system call. You can use this output to set shell variables using the eval builtin (page 374).

$ stat -s /
st_dev=234881026 st_ino=2 st_mode=041775 st_nlink=41 st_uid=0 st_gid=80
st_rdev=0 st_size=1496 st_atime=1121834492 st_mtime=1121660465
st_ctime=1121660465 st_blksize=4096 st_blocks=0 st_flags=0
$ eval $(stat -s /)
$ echo $st_uid
0

The st_uid variable indicates the file's owner as a numeric user ID; 0 is the user ID for root.

The following examples show different ways of displaying information about standard input. The first command displays the output in stat's default format. The second command displays the permissions (%p, the same as st_mode above) field using various formats. The S flag displays data as a string, not as a number. The H and M flags display the high and middle components of the value, respectively.

$ stat
31090580 32514436 crw--w---- 1 zach tty 67108867 0 "May  9 16:17:23 2005"
"May  9 16:17:23 2005" "May  9 16:17:23 2005" 131072 0 0 (stdin)

$ stat -f "p: %p Sp: %Sp Hp: %Hp SHp: %SHp Mp: %Mp SMp: %SMp"
p: 20620 Sp: crw--w---- Hp: 2 SHp: rw- Mp: 0 SMp: -w-

The high/middle/low components are applicable only to some fields and may select different parts of a field in different formats. For example, the high component of permissions in string form is the user permissions bits; in numeric form it is the file's type bit.

strings: Displays strings of printable characters

strings [options] file-list

The strings utility displays strings of printable characters from object and other nontext files.

Arguments

The file-list specifies the pathnames of one or more files that strings processes.

Options

a | (all) Searches all sections of object files. Without this option, strings processes only the initialized and loaded parts of an object file.
 | (this option is a hyphen) Searches entire files. This is the default behavior except when processing object files. By default string processes only relevant parts of object files.
n min
 | (minimum) Displays strings of characters that are at least min characters long (the default is 4). Also min.

Discussion

The strings utility can help you determine the contents of nontext files. One application for strings is determining the owner of files in a lost+found directory.

Examples

The following example displays the strings of four or more printable characters found in the executable file for the man utility. If you did not know what this file was, these strings could help you determine that it was the man executable.

$ strings /usr/bin/man
...
LC_CTYPE
APROPOS
%s %Q
WHATIS
MAN_HP_DIREXT
MAN_IRIX_CATNAMES
manpath
sman
%s/%s%s%s/%s.%s%s
%s/cat%s%s/%s.%s%s
/man
/MAN
/share/man
/man1
/man8
...

stty: Displays or sets terminal parameters

stty [options] [arguments]

Without any arguments, stty displays certain parameters affecting the operation of the terminal/terminal emulator. For a list of some of these parameters and an explanation of each, see "Arguments." The arguments establish or change parameters.

Options

a | (all) Reports on all parameters. This option does not accept arguments.
e | (everything) Reports on all parameters in a slightly different format from that produced by the a option. The command stty all generates the same output.
f /dev/device
 | (file) Affects device. Without this option, stty affects the device attached to standard input. You can change the characteristics of a device only if you own its device file or if you are Superuser.
g | (generate) Generates a report of the current settings in a format you can use as arguments to another stty command. This option does not accept arguments.

Arguments

The arguments to stty specify which terminal parameters stty is to alter. Turn on each of the parameters that is preceded by an optional hyphen (indicated in the following list as []) by specifying the parameter without the hyphen. Turn it off by using the hyphen. Unless specified otherwise, this section describes the parameters in their on states.

Special Keys and Characteristics

columns n

Sets the line width to n columns.

ek

(erase kill) Sets the erase and line kill keys to their default values. Many systems use DELETE and CONTROL-U, respectively, as the defaults.

erase x

Sets the erase key to x. To specify a control character, precede x with CONTROL-V (for example, use CONTROL-V CONTROL-H to indicate CONTROL-H) or use the notation ^h, where ^ is a caret (SHIFT-6 on most keyboards).

intr x

Sets the interrupt key to x. See erase x for conventions.

kill x

Sets the line kill key to x. See erase x for conventions.

rows n

Sets the number of screen rows to n.

sane

Sets the terminal parameters to values that are usually acceptable. The sane argument is useful when several stty parameters have changed, making it difficult to use the terminal to run stty to set things right. If sane does not appear to work, try entering the following characters:

CONTROL-J stty sane CONTROL-J

susp x

(suspend) Sets the suspend (terminal stop) key to x. See erase x for conventions.

werase x

(word erase) Sets the word erase key to x. See erase x for conventions. Some programs use CONTROL-W as the word erase key regardless of how you set werase.

Modes of Data Transmission

[]cooked

See raw.

[]cstopb

(stop bits) Selects two stop bits (cstopb specifies one stop bit).

[]parenb

(parity enable) Enables parity generation and checking on input and output. When you specify parenb, the system does not generate or expect a parity bit when communicating with the terminal.

[]parodd

(parity odd) Selects odd parity (parodd selects even parity).

[]raw

The normal state is raw. When the system reads input in its raw form, it does not interpret the following special characters: erase (usually DELETE), line kill (usually CONTROL-U), interrupt execution (CONTROL-C), and EOF (CONTROL-D). In addition, the system does not use parity bits. Reflecting the humor that is typical of UNIX's heritage, you can specify raw as cooked.

Treatment of Characters

[]echo

Echoes characters as they are typed (full-duplex operation). If a terminal is half-duplex and displays two characters for each one it should display, turn the echo parameter off (echo). Use echo when the user is entering a password.

[]echoe

(echo erase) The normal setting is echoe, which causes the kernel to echo the character sequence BACKSPACE SPACE BACKSPACE when you use the erase key to delete a character. The effect is to move the cursor backward across the line, removing characters as you delete them.

[]echoke

(echo kill erase) The normal setting is echoke. When you use the kill character to delete a line while this option is set, all characters back to the prompt are erased on the current line. When this option is negated, pressing the kill key moves the cursor to the beginning of the next line.

[]nl

Accepts only a NEWLINE character as a line terminator. With nl in effect, the system accepts a RETURN character from the terminal as a NEWLINE but sends a RETURN followed by a NEWLINE to the terminal instead of a NEWLINE.

[]tabs

Transmits each TAB character to the terminal as a TAB character. When tabs is turned off (tabs), the kernel translates each TAB character into the appropriate number of SPACEs and transmits them to the terminal.

Job Control Parameters

[]tostop

Stops background jobs if they attempt to send output to the terminal (tostop allows background jobs to send output to the terminal).

Notes

The name stty is an abbreviation for set teletypewriter, or set tty (page 882), the first terminal that UNIX was run on. Today stty is commonly thought of as set terminal.

The shells retain some control over standard input when you use them interactively. As a consequence, a number of the options available with stty appear to have no effect. For example, the command stty echo appears to have no effect under tcsh:

tcsh $ stty -echo
tcsh $ date
Fri Feb 18 21:21:14 PST 2005

While stty echo does work when you are using bash interactively, stty echoe does not. However, you can still use these options to affect shell scripts and other utilities.

$ cat testit
#!/bin/bash
stty -echo
echo -n "Enter a value: "
read a
echo
echo "You entered: $a"
stty echo

$ testit
Enter a value:
You entered: 77

In the preceding example, the kernel does not display the user's response to the Enter a value: prompt. The value is retained by the a variable and is displayed by the echo "You entered: $a" statement.

Examples

The first example shows that stty without any arguments displays several terminal operation parameters. (Your system may display more or different parameters.) The character following erase = is the erase key. A ^ preceding a character indicates a CONTROL key. In the example, the erase key is CONTROL-H. If stty does not display the erase character, it is set to its default value of DELETE. If you do not see a kill character, it is set to its default of CONTROL-U.

$ stty
speed 38400 baud;
lflags: echoe echoke echoctl pendin
oflags: onocr onlret
cflags: cs8 -parenb
erase
^H

Next the ek argument returns the erase and line kill keys to their default values:

$ stty ek

The next example verifies the change. The stty utility does not display either the erase character or the line kill character, indicating that both are set to their default values:

$ stty
speed 38400 baud;
lflags: echoe echoke echoctl pendin
oflags: onocr onlret
cflags: cs8 -parenb

The next example sets the erase key to CONTROL-H. The CONTROL-V quotes the CONTROL-H so that the shell does not interpret it and passes it to stty:

$ stty erase CONTROLV CONTROLH
$ stty
speed 38400 baud;
...
erase
^H

Next stty sets the line kill key to CONTROL-X. This time the user entered a caret (^) followed by an x to represent CONTROL-X. You can use either a lowercase or an uppercase letter.

$ stty kill ^X
$ stty
speed 38400 baud;
...
erase   kill
^H      ^X

Now stty changes the interrupt key to CONTROL-C:

$ stty intr CONTROLVCONTROLC

In the following example, stty turns off TABs so that the appropriate number of SPACEs are sent to the terminal in place of a TAB. Use this command if a terminal does not automatically expand TABs.

$ stty -tabs

sysctl: Displays and alters kernel variables

sysctl [options] [variable-list]

sysctl [options] w [var=value ... ]

The sysctl utility displays and alters kernel variables including kernel tuning parameters.

Arguments

The variable-list is a list of kernel variables whose values sysctl displays. In the second format, each value is assigned to the variable named var.

Options

a | (all) Displays all kernel variables.
b | (binary) Displays kernel variables as binary data without terminating NEWLINEs.
n | (no label) Displays variables without labels.

Discussion

The sysctl utility provides access to a number of kernel variables, including the maximum number of processes that the kernel will run at one time, the filename used for core files, and the system security level. Some variables cannot be altered or can be altered only in certain ways. For example, the security level can never be lowered.

Notes

During startup, /etc/rc uses sysctl to alter any variables specified in /etc/sysctl.conf, if it exists.

Examples

One common use of sysctl is tuning the kernel. The default limit on the number of processes running with a given user ID is 100 under Mac OS X. (It is higher in Mac OS X Server.) The process limits can be displayed without root privileges, but can be altered only by root. In this example, Alex changes this limit using sudo to gain root privileges temporarily:

$ sysctl -a | grep proc
kern.maxproc = 532
kern.maxfilesperproc = 10240
kern.maxprocperuid = 100
kern.aioprocmax = 16
kern.proc_low_pri_io = 0
$ sudo sysctl -w kern.maxprocperuid=200
Password:
kern.maxprocperuid: 100 -> 200
$

tail: Displays the last part of a file

tail [options] [file-list]

The tail utility displays the last part, or end, of a file.

Arguments

The file-list specifies the pathnames of one or more files that tail displays. When you specify more than one file, tail displays the filename of each file before displaying the lines of the file. If you do not specify an argument or if you specify a hyphen (-) instead of a filename, tail reads from standard input.

Options

c [+]n
 | (characters) Counts by bytes (characters) instead of lines. The n is an integer that specifies the number of bytes. Thus the command tail c 5 displays the last 5 bytes of a file.
 | If you put a plus sign (+) in front of n, tail counts from the start of the file instead of the end. The tail utility still displays to the end of the file, even though it starts counting from the beginning. Thus tail c +5 causes tail to display from the fifth character through the end of the file.
f | (flush) After copying the last line of the file, tail enters an endless loop, waiting and copying additional lines from the file if the file grows. If you specify multiple files in file-list with this option, tail includes a new header each time it displays output from a different file so that you know which file is being added to. This option is useful for tracking the progress of a process that is running in the background and sending its output to a file. The tail utility continues to wait indefinitely, so you must use the interrupt key to terminate it.
n [+]n
 | (lines) Counts by lines (the default). The n is an integer that specifies the number of lines. Although it is not documented, you can use ±n to specify a number of lines without using this option.
 | If you put a plus sign (+) in front of n, tail counts from the beginning of the file instead of the end. The tail utility still displays to the end of the file, even though it starts counting from the beginning. Thus tail n +5 causes tail to display from the fifth line through the last line of the file.

Notes

The tail utility displays the last ten lines of its input by default.

Examples

The examples are based on the eleven file:

$ cat eleven
line one
line two
line three
line four
line five
line six
line seven
line eight
line nine
line ten
line eleven

First tail displays the last ten lines of the eleven file (no options):

$ tail eleven
line two
line three
line four
line five
line six
line seven
line eight
line nine
line ten
line eleven

Next it displays the last three lines (n 3) of the file:

$ tail -n 3 eleven
line nine
line ten
line eleven

The following example displays the file starting at line 8 (+8):

$ tail -n +8 eleven
line eight
line nine
line ten
line eleven

The next example displays the last six characters in the file (c 6). Only five characters are evident (leven); the sixth is a NEWLINE.

$ tail -c 6 eleven
leven

The final example demonstrates the f option. Here tail tracks the output of a make command, which is being sent to the file accounts.out:

$ make accounts > accounts.out &
$ tail -f accounts.out
        cc -c trans.c
        cc -c reports.c
...
CONTROL-C
$

In the preceding example, using tail with f has the same effect as running make in the foreground and letting its output go to the terminal. However, using tail offers some advantages. First, the output of make is saved in a file. (The output would not be saved if you let it go to the terminal.) Second, if you decide to do something else while make is running, you can kill tail and use the screen while make continues in the background. When you are running a large job, such as compiling a large program, you can use tail with the f option to check its progress periodically.

tar: Stores or retrieves files to/from an archive file

tar option [modifiers] [file-list]

The tar (tape archive) utility creates, adds to, lists, and retrieves files from an archive file.

Arguments

The file-list specifies the pathnames of one or more files that tar creates an archive with, adds to an archive, or extracts from an archive.

Options

Use only one of the following options to indicate which action tar should take. You can alter the action of the option by following it with one or more modifiers.

create | c | Creates an archive. This option stores the files named in file-list in a new archive. If the archive already exists, it is deleted before the new archive is created. If the file-list is a directory, tar recursively copies the files within the directory into the archive. Without the file option, the archive is written to standard output.
compare | d | (diff) Compares an archive with the corresponding disk files and reports on the differences.
help |  | Displays a list of options and modifiers, with short descriptions of each.
append | r | Writes the files named in file-list to the end of the archive. This option leaves files that are already in the archive intact, so duplicate copies of files may appear in the archive when tar finishes. When tar extracts the files, the most recent copy of a file in the archive ends up on the disk.
list | t | (table of contents) Without a file-list, this option produces a table of contents listing all files in an archive. With a file-list, it displays the name of each file in the file-list each time it occurs in the archive. You can use this option with the verbose option to display detailed information about each file in an archive. Without the file option, the archive is read from standard input.
update | u | Adds the files from file-list if they are not already in the archive or if they have been modified since they were last written to the archive. Because of the additional checking required, tar runs more slowly when you specify this option.
extract | x | Extracts file-list from the archive and writes it to the disk. Overwrites existing files with the same names. Without a file-list, this option extracts all files from the archive. If file-list includes a directory, tar extracts that directory hierarchy. The tar utility attempts to keep the owner, modification time, and access privileges the same as those associated with the original files. If it reads the same file more than once, the last version read will appear on the disk when tar is finished. Without the file option, the archive is read from standard input.

Modifiers

You can specify one or more modifiers following an option. If you use the single-character form of the modifier, a leading hyphen is not required, although it is a good practice to use it.

If a modifier takes an argument, that modifier must be the last one in a group. For example, the arguments are arranged legally in the following tar command:

$ tar -cb 10 -f memos.tar memos

Conversely, the following tar command generates an error:

$ tar -cbf 10 memos.tar memos
tar: f: Invalid blocking factor
Try 'tar --help' for more information.

The error occurs because the b modifier takes an argument but is not the last modifier in a group.

blocking-factor=n

b n

Uses n as the blocking factor for creating an archive. Use this option only when tar is creating an archive directly to a tape. (When tar reads a tape archive, it automatically determines the blocking factor.) The value of n is the number of 512-byte blocks to write as a single block on the tape. Apple does not supply a tape driver for OS X, but third-party tape drivers are available.

directory=dir

C dir

Changes the working directory to dir before processing.

checkpoint

Displays periodic messages. This option lets you know tar is running without displaying all the verbose messages.

file=filename

f filename

Uses filename as the name of the file (device) that holds the archive. The filename can be the name of an ordinary file or a device (such as a tape drive). You can use a hyphen (-) instead of the filename to refer to standard input when creating an archive and to refer to standard output when extracting files from an archive. The following two commands are equivalent ways of creating a compressed archive of the files under the /Users directory in the /tmp directory:

$ tar -zcf /tmp/Users.tgz /Users
$ tar -cf - /Users | gzip > /tmp/Users.tgz

dereference

h

Archives the files that symbolic links point to, not the links themselves.

exclude=file

Does not process the file named file. If file is a directory, no files or directories within that directory hierarchy are processed. The file can be an ambiguous file reference; quote special characters as needed.

ignore-failed-read

When creating an archive, tar normally quits with a nonzero exit status if any file in file-list is unreadable. This option causes tar to continue processing, skipping unreadable files.

bzip

j

Uses bzip2 (page 668) to compress/decompress files when creating an archive and extracting files from an archive.

one-file-system

l

(lowercase "l") When a directory name appears in file-list while tar is creating an archive, tar recursively processes the files and directories below the named directory. With this option, tar stays in the filesystem that contains the named directory and does not process directories in other filesystems.

tapelength=n

L n

Asks for a new tape after writing n*1,024 bytes to the current tape. This feature is useful when you are building archives that are too large to fit on a single tape.

touch

m

Sets the modification time of an extracted file to the time of extraction. Without this option, tar attempts to maintain the modification time of the original file.

absolute-paths

P

The default behavior of tar is to force all pathnames to be relative paths by stripping leading slashes. This option disables this feature, so absolute pathnames remain as absolute paths.

sparse

S

UNIX allows you to have sparse filesthat is, large, mostly empty fileson disk. The empty sections of sparse files do not take up any disk space. When tar copies a sparse file out of an archive, it normally expands the file to its full size. As a result, when you restore a sparse file from a tar backup, the file takes up its full space and may no longer fit in the same disk space as the original. This option causes tar to handle sparse files efficiently so that they do not take up unnecessary space either in the archive or when they are extracted.

verbose

v

Lists each file as tar reads or writes it. When combined with the t option, v causes tar to display a more detailed listing of the files in an archive, showing their ownership, permissions, size, and other information.

interactive or confirmation

w

Asks you for confirmation before reading or writing each file. Respond with y if you want tar to take the action. Any other response causes tar not to take the action.

exclude-from=filename

X filename

Similar to the exclude option except that filename specifies a file that contains a list of files to exclude from processing. Each file listed in filename must appear on a separate line.

gzip or gunzip

z

Causes tar to use gzip (page 756) to compress an archive while it is being created and to use gunzip to decompress an archive when extracting files from it. This option also works to extract files from archives that have been compressed with the compress utility.

compress or uncompress

Z

Uses compress when creating an archive and uncompress when extracting files from an archive.

Notes

The help option displays all of the tar options and modifiers. The info page on tar provides extensive information, including a tutorial.

You can use ambiguous file references in file-list when you create an archive but not when you extract files from an archive.

The name of a directory file within the file-list references all files and subdirectories within that directory.

The file that tar sends its output to by default is compilation specific; typically it goes to standard output. Use the f option to specify a different filename or device to hold the archive.

When you create an archive using a simple filename in file-list, the file appears in the working directory when you extract it. If you use a relative pathname when you create an archive, the file appears with that relative pathname, starting from the working directory when you extract it. If you use the P option and an absolute pathname when you create an archive, and then use the P option when you extract files from the archive, tar extracts the file with the same pathname.

Under Mac OS X version 10.4 and later, tar archives extended attributes (page 93), including ACLs. Under version 10.3 and earlier, tar does not archive extended attributes. For a similar utility that does copy extended attributes, see ditto (page 715).

Examples

The first example makes a copy of the /Users/alex directory hierarchy in an archive file named alex.tar on a volume (filesystem) named Backups. The v modifier causes the command to list the files it writes to the archive. The message from tar explains that the default action is to store all pathnames as relative paths instead of as absolute paths, thereby allowing you to extract the files into a different directory on the disk.

$ tar -cvf /Volumes/Backups/alex.tar /Users/alex
tar: Removing leading '/' from member names.
Users/alex/
Users/alex/.bash_history
Users/alex/.bash_profile
...

In the next example, the same directory is saved to an archive on the partition /Volumes/Firewire. Without the v modifier, tar does not display the list of files it is writing to the archive. The command runs in the background and displays any messages after the shell issues a new prompt.

$ tar -c -f /Volumes/Firewire /Users/alex &
[1] 4298
$ tar: Removing leading '/' from member names.

The next command displays the table of contents of the archive file alex.tar:

$ tar -tvf /Volumes/Backups/alex.tar
drwxrwxrwx alex/alex        0 Jun 30 21:39 2004 Users/alex/
-rw-r--r-- alex/alex      678 Aug  6 14:12 2005 Users/alex/.bash_history
-rw-r--r-- alex/alex      571 Aug  6 14:06 2005 Users/alex/.bash_profile
drwx------ alex/alex        0 Nov  6 22:34 2005 Users/alex/mail/
-rw------- alex/alex     2799 Nov  6 22:34 2005 Users/alex/mail/sent-mail
...

In the following example, Alex creates a gzipped tar archive in /tmp/alex.tgz. This approach is a popular way to bundle files that you want to transfer over a network or otherwise share with others. Ending a filename with .tgz is one convention for identifying gzipped tar archives. Another convention is to end the filename with .tar.gz.

$ tar -czf /tmp/alex.tgz literature

The final command lists the files in the compressed archive alex.tgz:

$ tar -tzvf /tmp/alex.tgz
...

tee: Copies standard input to standard output and one or more files

tee [options] file-list

The tee utility copies standard input to standard output and to one or more files.

Arguments

The file-list specifies the pathnames of one or more files that receive output from tee.

Options

Without any options, tee overwrites the output files if they exist and responds to interrupts. If a file in file-list does not exist, tee creates it.

a | (append) Appends output to existing files rather than overwriting them.
i | (ignore) Causes tee not to respond to interrupts.

Examples

In the following example, a pipe sends the output from make to tee, which copies it to standard output and the file accounts.out. The copy that goes to standard output appears on the screen. The cat utility displays the copy that was sent to the file.

$ make accounts | tee accounts.out
        cc -c trans.c
        cc -c reports.c
...
$ cat accounts.out
        cc -c trans.c
        cc -c reports.c
...

Refer to page 861 for a similar example that uses tail f rather than tee.

telnet: Connects to a remote system over a network

telnet [options] [remote-system [port]]

The telnet utility implements the TELNET protocol to connect to a remote system over a network. It can use the TELNET protocol or it can connect to other ports and provide direct access to other protocols.

Security: telnet is not secure

The telnet utility is not secure. It sends your username and password over the network as cleartext, which is not a secure practice. Use ssh (page 847) when it is available.

Arguments

The remote-system is the name or IP address of the remote system that telnet connects to. When you do not specify a remote-system, telnet works interactively and prompts you to enter one of the commands described under "Discussion." The port is the port on the remote-system that telnet connects to. When you do not specify a port, telnet connects to port 23 and attempts to log in.

Options

e | c
 | (escape) Changes the escape character from CONTROL-] to the character c.
K | (no login) Prevents automatic login.
l | username
 | (login) Attempts an automatic login on the remote system using username. If the remote system understands how to handle automatic login with telnet, you are prompted for a password.

Discussion

After telnet connects to a remote system, you can put telnet in command mode by typing the escape character (usually CONTROL-]). A remote system should report the escape character it recognizes. To leave command mode, press RETURN on a line by itself.

In command mode, telnet displays the telnet> prompt. You can use the following commands in command mode:

? | (help) Displays a list of commands recognized by the telnet utility on the local system.
close | Closes the connection to the remote system. If you specified the name of a system on the command line when you started telnet, close has the same effect as quit: The telnet program terminates, and the shell displays a prompt. If you used the open command instead of specifying a remote system on the command line, close returns telnet to command mode.
logout | Logs you out of the remote system; similar to close.
open remote-system
 | If you did not specify a remote-system on the command line or if the attempt to connect to the system failed, you can specify the name of a remote-system interactively with the open command.
quit | Quits the telnet session.
z | Suspends the telnet session. When you suspend a session, you return to the login shell on the local system. To resume the suspended telnet session, type fg at a shell prompt. For more information refer to "Moving a Job from the Foreground to the Background" on page 132.

Notes

Many computers, including some non-UNIX-like systems and nearly all UNIX-like systems, support the TELNET protocol. The telnet utility is a user interface to this protocol for OS X that allows you to connect to many different types of systems. Although you typically use telnet to log in, the remote computer may offer other services through telnet, such as access to special databases. Often these services are reached by specifying a port number. See page 405 for more information.

Examples

In the first example, the user connects to the remote system named bravo. After running a few commands, the user escapes to command mode and uses the z command to suspend the telnet session so as to run a few commands on the local system. The user gives an fg command to the shell to resume using telnet. The logout command on the remote system ends the telnet session, and the local shell displays a prompt.

kudos $ telnet bravo
Trying 192.168.0.55 ...
Connected to bravo.
Escape character is '^]'.

Fedora Core Release 2 (Tettnang)
Kernel 2.6.5-1.358 on an i686

login: watson
Password:
Last login: Wed Jul 31 10:37:16 from kudos
bravo $
...
bravo $CONTROL-]
telnet> z

[1]+ Stopped  telnet bravo
kudos $
...
kudos $ fg
telnet bravo

bravo $ logout
Connection closed by foreign host.
kudos $

In the next example, the user connects manually to the SMTP server at example.com to see why it is bouncing mail from the spammer.com domain. The first line of output indicates which IP address telnet is trying to connect to. After telnet displays the Connected to smtpserv.example.com message, the user emulates an SMTP dialog, following the standard SMTP protocol. The first line, which starts with helo, begins the session and identifies the local system. After the SMTP server responds, the user enters a line that identifies the mail sender as user@spammer.com. The SMTP server's response explains why the message is bouncing, so the user ends the session with quit.

$ telnet smtpserv 25
Trying 192.168.1.1...
Connected to smtpserv.example.com.
Escape character is '^]'.
helo example.com
220 smtpserv.example.com ESMTP Sendmail 8.13.1/8.13.1; Wed, 11 May 2005 00:13:43 -0500 (CDT)
250 smtpserv.example.com Hello desktop.example.com [192.168.1.97], pleased to meet you
mail from:user@spammer.com
571 5.0.0 Domain banned for spamming
quit
221 2.0.0 smtpserv.example.com closing connection

The telnet utility allows you to use any protocol you want, as long as you know it well enough to type it manually.

test: Evaluates an expression

test expression

[ expression ]

The test utility evaluates an expression and returns a condition code indicating that the expression is either true (0) or false (not 0). You can place brackets ([]) around the expression instead of using the word test (second format).

Arguments

The expression contains one or more criteria (Table VI-27) that test evaluates. A a separating two criteria is a Boolean AND operator: Both criteria must be true for test to return a condition code of true. A o is a Boolean OR operator: one or the other (or both) of the criteria must be true for test to return a condition code of true.

Table VI-27. Criteria

Criterion

Meaning

string

True if string is not a null string.

n string

True if string has a length greater than zero.

z string

True if string has a length of zero.

string1 = string2

True if string1 is equal to string2.

string1 != string2

True if string1 is not equal to string2.

int1 relop int2

True if integer int1 has the specified algebraic relationship to integer int2. The relop is a relational operator from Table VI-28. As a special case, l string, which gives the length of string, may be used for int1 or int2.

file1ef file2

True if file1 and file2 have the same device and inode numbers.

file1 nt file2

True if file1 was modified after file2 (the modification time of file1 is newer than that of file2).

file1 ot file2

True if file1 was modified before file2 (the modification time of file1 is older than that of file2).

b filename

True if the file named filename exists and is a block special file.

c filename

True if the file named filename exists and is a character special file.

d filename

True if the file named filename exists and is a directory.

e filename

True if the file named filename exists.

f filename

True if the file named filename exists and is an ordinary file.

g filename

True if the file named filename exists and its setgid bit (page 90) is set.

G filename

True if the file named filename exists and is associated with the group that is the primary group of the user running the command (same effective group ID).

k filename

True if the file named filename exists and its sticky bit (page 955) is set.

L filename

True if the file named filename exists and is a symbolic link.

O filename

True if the file named filename exists and is owned by the user running the command (same effective user ID).

p filename

True if the file named filename exists and is a named pipe.

r filename

True if the file named filename exists and you have read permission for it.

s filename

True if the file named filename exists and contains information (has a size greater than 0 bytes).

t file-descriptor

True if file-descriptor is associated with the screen/keyboard. The file-descriptor for standard input is 0, for standard output is 1, and for standard error is 2.

u filename

True if the file named filename exists and its setuid bit (page 90) is set.

w filename

True if the file named filename exists and you have write permission for it.

x filename

True if the file named filename exists and you have execute permission for it.

Table VI-28. Relational operators

Relop

Meaning

eq

Equal to

ge

Greater than or equal to

gt

Greater than

le

Less than or equal to

lt

Less than

ne

Not equal to

You can negate a criterion by preceding it with an exclamation point (!). You can group criteria with parentheses. If there are no parentheses, a takes precedence over o, and test evaluates operators of equal precedence from left to right.

Within the expression you must quote special characters, such as parentheses, so that the shell does not interpret them but rather passes them unchanged to test.

Because each element, such as a criterion, string, or variable within the expression, is a separate argument, you must separate each element from other elements with a SPACE. Table VI-27 lists the criteria you can use within the expression. Table VI-28 lists test's relational operators.

Notes

The test command is built into the Bourne Again and TC Shells. There is also a stand-alone test utility in /bin and a hard link to the file named /bin/[.

Examples

The following examples demonstrate the use of the test utility in Bourne Again Shell scripts. Although test works from the command line, it is more commonly employed in shell scripts to test input or verify access to a file.

The first example prompts the user, reads a line of input into a variable, and uses the synonym for test, [], to see whether the user entered yes:

$ cat user_in
echo -n "Input yes or no: "
read user_input
if [ "$user_input" = "yes" ]
    then
       echo You input yes.
fi

The next example prompts for a filename and then uses the synonym for test, [], to see whether the user has read access permission (r) for the file and (a) whether the file contains information (s):

$ cat validate
echo -n "Enter filename: "
read filename
if [ -r "$filename" -a -s "$filename" ]
    then
        echo File $filename exists and contains information.
        echo You have read access permission to the file.
fi

The t 1 criterion checks whether the process running test is sending standard output to the screen. If it is, the test utility returns a value of true (0). The shell stores the exit status of the last command it ran in the $? variable. The following script tests whether its output is goes to a terminal:

$ cat term
test -t 1
echo "This program is (=0) or is not (=1)
sending its output to a terminal:" $?

First term is run with the output going to the terminal:

$ term
This program is (=0) or is not (=1)
sending its output to a terminal: 0

The next example runs term and redirects the output to a file. The contents of the file temp show that test returned 1, indicating that its output did not go to a terminal.

$ term > temp
$ cat temp
This program is (=0) or is not (=1)
sending its output to a terminal: 1

top: Dynamically displays process status

top [options]

The top utility displays information about the status of the local system, including information about the current processes. Many of the options and configuration commands work in compatibility mode only (see the X option); this section describes that mode.

Options

Although top does not require the use of hyphens with options, it is a good idea to include them for clarity and consistency with other utilities. You can cause top to run as though you had specified any of the options by giving commands to the utility while it is running. See "Discussion" for more information.

n n
 | (number) Displays n processes.
o key
 | (order) Sorts the display according to key, which may be cpu, uid, time, pid, or another key. The default sort key is pid.
ssecs
 | (sleep) Causes top to redisplay information every secs seconds. The default is 1.
X | (compatibility) Causes top to run in compatibility mode. This mode is the most useful.

Discussion

The first few lines that top displays summarize the status of the local system. The exact lines that top displays vary depending on which options you use; see the top man page for details. The display at the top of the screen provides details about the system load average and uptime, processor usage, memory usage, and overall statistics about processes and threads.

The rest of the display reports on individual processes, listing the most CPU-intensive processes first. By default top displays the number of processes that fit on the screen. Table VI-29 describes the meanings of the fields displayed for each process. Options can change which fields top displays.

Table VI-29. Fields displayed by top

Name

Meaning

PID

Process identification number

UID

User ID of the process

COMMAND

Command line that started the process or name of the program

TIME

Total CPU time used by the process

%CPU

Percentage of CPU time used by the process during the last update period

#TH

Number of threads in the process

#PRTS

Number of Mach ports

#MREGS

Number of memory regions

RPRVT

Resident private memory size

RSHRD

Resident shared memory size

RSIZE

Total resident memory size, including shared pages

VSIZE

Total address space allocated, including shared pages

While top is running, you can use the following commands to modify its behavior. Except as noted, top must be in compatibility mode (X option) for these commands to work. Commands must be terminated by pressing RETURN.

n | (number) Prompts for the number of processes to display. If you enter 0 (the default), top displays as many processes as will fit on the screen.
q | (quit) Terminates top (works from all modes).
r | Toggles the display of process memory maps (RPRVT, RSHRD, VPRVT).
s | (seconds) Prompts you for the number of seconds to delay between updates to the display (3 is the default). You may enter an integer, a fraction, or 0 (for continuous updates). This command is disabled in secure mode.
S | (signal) Sends a signal to a process. The top utility prompts for a signal (the default is TERM) and then for a PID number.
SPACE
 | Refreshes the screen (works from all modes). Any key that does not do something else may refresh the screen.

Notes

The top utility is similar to ps but updates the display periodically, enabling you to watch the behavior of the local system over time.

This utility shows only as much of the command line for each process as fits on a line. When a process is swapped out, top replaces the command line with the name of the command in parentheses.

Requesting continuous updates is almost always a mistake. The display updates too quickly and the system load increases dramatically.

By default top sorts by PID number. To display processes with the one using the most CPU time at the top, give a top o cpu command.

Examples

The following display is the result of a typical execution of top X:

Time: 2005/05/11 00:36:55. Threads: 153. Procs: 51, 2 running, 49 sleeping.
LoadAvg:  0.02,  0.03,  0.00.  CPU:   0.0% user,   4.8% sys,  95.2% idle.
SharedLibs: num =  148, resident = 24.9M code, 3.09M data, 5.91M linkedit.
MemRegions: num =  4139, resident = 48.4M + 5.66M private, 38.8M shared.
PhysMem: 65.1M wired, 65.3M active,  101M inactive,  232M used,  791M free.
VirtMem: 3.83G + 88.7M,      12920 pageins,          0 pageouts.

PID  UID  REG RPRVT  RSHRD  RSIZE  VPRVT   VSIZE  TH PRT    TIME  %CPU COMMAND
2971   0  22   524K   396K   976K  26.2M   26.9M   1  19   0.77s   4.6 top
2889 501  18   140K   424K   480K  26.2M   26.7M   1  13   0.02s   0.0 less
2886 501  16  76.0K   812K   364K  1.05M   27.1M   1   8   0.00s   0.0 sh
2885 501  16  72.0K   812K   560K  1.05M   27.1M   1  13   0.00s   0.0 sh
2884 501  16   168K   360K   440K  26.2M   26.6M   1  13   0.01s   0.0 man
1761 501  18   232K   844K   876K  26.2M   27.1M   1  14   0.21s   0.0 bash
1535 501  17   184K   844K   820K  26.1M   27.1M   1  14   0.06s   0.0 bash
1531 501  41   100K  1.54M   512K   100K   29.9M   1  11   0.20s   0.0 sshd
1529   0  42   132K  1.54M  1.34M   136K   30.0M   1  34   0.15s   0.0 sshd
1521 501  16   188K   844K   812K  26.1M   27.1M   1  14   0.09s   0.0 bash
1520   0  36   144K   412K   556K  26.3M   26.9M   1  16   0.01s   0.0 login
235  501  16   192K   844K   828K  26.1M   27.1M   1  14   0.30s   0.0 bash
234    0  36   144K   412K   556K  26.3M   26.9M   1  16   0.01s   0.0 login

touch: Changes a file's access and/or modification time

touch [options] file-list

The touch utility changes the access and/or modification time of a file to the current time or a time you specify. If the file does not exist, touch creates it.

Arguments

The file-list specifies the pathnames of one or more files that touch will update.

Options

Accepts the common options described on page 663. Without any options, touch changes the access and modification times to the current time. When you do not specify the c option, touch creates files that do not exist.

a

(access) Updates the access time only, leaving the modification time unchanged.

c

(no create) Does not create files that do not exist.

m

(modification) Updates the modification time only, leaving the access time unchanged.

r file

(reference) Updates times to the times of file.

t [[cc]yy]nnddhhmm[.ss]

(time) Changes times to the date specified by the argument. The nn is the number of the month (0112), dd is the day of the month (0131), hh is the hour based on a 24-hour clock (0023), and mm is the minutes (0059). You must specify at least these fields. You can specify the number of seconds past the start of the minute with .ss.

The optional cc specifies the first two digits of the year (the value of the century minus 1), and yy specifies the last two digits of the year. When you do not specify a year, touch assumes the current year. When you do not specify cc, touch assumes 20 for yy in the range 068 and 19 for yy in the range 6999.

Examples

The first three commands show touch updating an existing file. The ls utility with the l option displays the modification time of the file. The last three commands show touch creating a file.

$ ls -l program.c
-rw-r--r--   1 alex alex 5860 Apr 21 09:54 program.c
$ touch program.c
$ls -l program.c
-rw-r--r--   1 alex alex 5860 Aug 13 19:01 program.c

$ ls -l read.c
ls: read.c: No such file or directory
$ touch read.c
$ ls -l read.c
-rw-rw-r--   1 alex alex 0 Aug 13 19:01 read.c

The next example demonstrates the use of the a option to change the access time only and the t option to specify a date for touch to use instead of the current date and time. The first ls command displays the file modification times; the second ls (with the u option) displays the file access times. After the touch command is executed, ls shows that the access times of the files cases and excerpts have been updated as specified.

$ ls -l
-rw-rw-r--  1 alex alex 45 Nov 30  2005 cases
-rw-rw-rw-  1 alex alex 14 Jan 8  2006 excerpts
$ ls -l -u
-rw-rw-r--  1 alex alex 45 Jul 17 19:47 cases
-rw-rw-rw-  1 alex alex 14 Jul 17 19:47 excerpts

$ touch -a -t 07301900 cases excerpts

$ ls -l
-rw-rw-r--  1 alex alex 45 Nov 30  2005 cases
-rw-rw-rw-  1 alex alex 14 Jan 8  2006 excerpts
$ ls -l -u
-rw-rw-r--  1 alex alex 45 Jul 30 19:00 cases
-rw-rw-rw-  1 alex alex 14 Jul 30 19:00 excerpts

tr: Replaces specified characters

tr [options] string1 [string2]

The tr utility reads standard input and, for each input character, maps it to an alternate character, deletes the character, or leaves the character alone. This utility reads from standard input and writes to standard output.

Arguments

The tr utility is typically used with two arguments, string1 and string2. The position of each character in the two strings is important: Each time tr finds a character from string1 in its input, it replaces that character with the corresponding character from string2.

With one argument, string1, and the d option, tr deletes the characters specified in string1. The option s replaces multiple sequential occurrences of characters in string1 with single occurrences (for example, abbc becomes abc).

Ranges

A range of characters is similar in function to a character class within a regular expression (page 899). The tr utility does not support ranges (character classes) enclosed within brackets. You can specify a range of characters by following the character that appears earlier in the collating sequence with a hyphen and then the character that comes later in the collating sequence. For example, 16 expands to 123456. Although the range AZ expands as you would expect in ASCII, this approach does not work when you use the EBCDIC collating sequence, as these characters are not sequential in EBCDIC. See "Character Classes" for a solution to this issue.

Character Classes

A TR character class is not the same as the character classes described elsewhere in this book. (GNU documentation uses the term list operator for what this book calls a character class. See "Brackets" on page 899 for more information.) You specify a character class as '[:class:]', where class is a character class from Table VI-30. You must specify a character class in string1 unless you are performing case conversion (see "Examples") or are using the d and s options together.

Table VI-30. Character classes

Class

Meaning

alnum

Letters and digits

alpha

Letters

blank

Whitespace

cntrl

CONTROL characters

digit

Digits

graph

Printable characters but not SPACEs

lower

Lowercase letters

print

Printable characters including SPACEs

punct

Punctuation characters

space

Horizontal or vertical whitespace

upper

Uppercase letters

xdigit

Hexadecimal digits

Options

c | (complement) Complements string1, causing TR to match all characters except those in string1.
d | (delete) Deletes characters that match those specified in string1. If you use this option with the s option, you must specify both string1 and string2 (see "Notes").
s | (squeeze) Replaces multiple sequential occurrences of a character in string1 with a single occurrence of the character when you call TR with only one string argument. If you use both string1 and string2, the tr utility first translates the characters in string1 to those in string2; it then reduces multiple sequential occurrences of characters in string2.

Notes

When string1 is longer than string2, the initial portion of string1 (equal in length to string2) is used in the translation. When string1 is shorter than string2, tr uses the last character of string1 to extend string1 to the length of string2. Here tr departs from the POSIX standard, which does not define a result in this situation.

If you use the d and s options together, tr deletes the characters in string1 and then reduces multiple sequential occurrences of characters in string2.

Examples

You can use a hyphen to represent a range of characters in string1 or string2. Both command lines in the following example produce the same result:

$ echo abcdef | tr  'abcdef' 'xyzabc'
xyzabc
$ echo abcdef | tr  'a-f' 'x-za-c'
xyzabc

The next example demonstrates a popular method for disguising text, often called ROT13 (rotate 13) because it replaces the first letter of the alphabet with the thirteenth, the second with the fourteenth, and so forth.

$ echo The punchline of the joke is ... |
> tr 'A-M N-Z a-m n-z' 'N-Z A-M n-z a-m'
Gur chapuyvar bs gur wbxr vf ...

To make the text intelligible again, reverse the order of the arguments to tr:

$ echo Gur chapuyvar bs gur wbxr vf ... |
> tr 'N-Z A-M n-z a-m' 'A-M N-Z a-m n-z'
The punchline of the joke is ...

The d option causes tr to delete selected characters:

$ echo If you can read this, you can spot the missing vowels! |
> tr -d 'aeiou'
If y cn rd ths, y cn spt th mssng vwls!

In the following example, TR replaces characters and reduces pairs of identical characters to single characters:

$ echo tennessee | tr -s 'tnse' 'srne'
serene

The next example replaces each sequence of nonalphabetic characters (the complement of all the alphabetic characters as specified by the character class alpha) in the file draft1 with a single NEWLINE character. The output is a list of words, one per line.

$ tr -c -s '[:alpha:]' '\n' < draft1

The next example uses character classes to upshift the string hi there:

$ echo hi there | tr '[:lower:]' '[:upper:]'
HI THERE

tty: Displays the terminal pathname

tty [option]

The tty utility displays the pathname of standard input if it is a terminal and displays not a tty if it is not a terminal. The exit status is 0 if standard input is a terminal and 1 if it is not.

Options

s (silent) Causes tty not to print anything; sets the exit status.

Notes

The term tty is short for teletypewriter, the terminal device on which UNIX was first run. This command appears in UNIX, and OS X has retained it for the sake of consistency and tradition.

Examples

The following example illustrates the use of tty:

$ tty
/dev/ttyp0
$ echo $?
0
$ tty < memo
not a tty
$ echo $?
1

umask: Establishes the file-creation permissions mask

umask [mask]

The umask builtin specifies a mask that the system uses to set access permissions when you create a file. This builtin works slightly differently in each of the shells.

Arguments

The mask can be a three-digit octal number (bash and tcsh) or a symbolic value (bash) such as you would use with chmod (page 676). The mask specifies the permissions that are not allowed.

When mask is an octal number, the digits correspond to the permissions for the owner of the file, members of the group the file is associated with, and everyone else. Because the mask specifies the permissions that are not allowed, the system subtracts each of these digits from 7 when you create a file. The resulting three octal numbers specify the access permissions for the file (the numbers you would use with chmod). A mask that you give as a symbolic value also specifies the permissions that are not allowed. See "Notes."

Without any arguments, umask displays the file-creation permissions mask.

Notes

Most utilities and applications do not attempt to create files with execute permissions, regardless of the value of mask; they assume that you do not want an executable file. As a result, when a utility or application such as touch creates a file, the system subtracts each of the digits in mask from 6. The exception is mkdir, which assumes that you do want the execute (access in the case of a directory) bit set. See "Examples."

The umask program is a builtin in bash and tcsh and generally goes in the initialization file for your shell (~/.bash_profile [bash] or ~/.login [tcsh]).

Under bash the argument u=rwx,go=r turns off all bits in the mask for the owner and turns on the write bit in the mask for groups and other users (the mask is 0033), causing those bits to be on in file permissions (744 or 644). Refer to chmod on page 676 for more information about symbolic permissions.

Examples

The following commands set the file-creation permissions mask and display the mask and its effect when you create a file and a directory. The mask of 022, when subtracted from 777, gives permissions of 644 (rwrr) for a file and 755 (rwxrxrx) for a directory:

$ umask 022
$ umask
0022
$ touch afile
$ mkdir adirectory
$ ls -ld afile adirectory
drwxr-xr-x  2 max  max  4096 Jul 24 11:25 adirectory
-rw-r--r--  1 max  max  0 Jul 24 11:25 afile

The next example sets the same mask value symbolically:

$ umask u=rwx,g=rx,o=rx
$ umask
0022

uniq: Displays unique lines

uniq [options] [input-file] [output-file]

The uniq utility displays its input, removing all but one copy of successive repeated lines. If the file has been sorted (see sort on page 837), uniq ensures that no two lines that it displays are the same.

Arguments

When you do not specify the input-file, uniq reads from standard input. When you do not specify the output-file, uniq writes to standard output.

Options

A field is a sequence of characters bounded by SPACEs, TABs, NEWLINEs, or a combination of these characters.

c | (count) Precedes each line with the number of occurrences of the line in the input file.
d | (duplicates) Displays one copy of lines that are repeated; does not display lines that are not repeated.
f | nfield
 | (fields) Ignores the first nfield blank-separated fields of each line. The uniq utility bases its comparison on the remainder of the line, including the leading blanks of the next field on the line (see the s option).
s | nchar
 | (skip) Ignores the first nchar characters of each line. If you also use the skip-fields option, uniq ignores the first nfield fields followed by nchar characters. You can use this option to skip over leading blanks of a field.
u | (unique) Displays only lines that are not repeated.

Examples

These examples assume that the file named test in the working directory contains the following text:

$ cat test
boy took bat home
boy took bat home
girl took bat home
dog brought hat home
dog brought hat home
dog brought hat home

Without any options, uniq displays one copy of successive repeated lines:

$ uniq test
boy took bat home
girl took bat home
dog brought hat home

The c option displays the number of consecutive occurrences of each line in the file:

$ uniq -c test
   2 boy took bat home
   1 girl took bat home
   3 dog brought hat home

The d option displays only lines that are consecutively repeated in the file:

$ uniq -d test
boy took bat home
dog brought hat home

The u option displays only lines that are not consecutively repeated in the file:

$ uniq -u test
girl took bat home

Next the f option skips the first field in each line, causing the lines that begin with boy and the line that begins with girl to appear to be consecutive repeated lines. The uniq utility displays only one occurrence of these lines:

$ uniq -f 1 test
boy took bat home
dog brought hat home

The next example uses the f and s options first to skip two fields and then to skip two characters. The two characters this command skips include the SPACE that separates the second and third fields and the first character of the third field. Ignoring these characters, all the lines appear to be consecutive repeated lines containing the string at home. The uniq utility displays only the first of these lines:

$ uniq -f 2 -s 2 test
boy took bat home

w: Displays information about system users

w [options] [user-list]

The w utility displays the names of users who are currently logged in, together with their terminal device numbers, the times they logged in, the commands they are running, and other information.

Arguments

The user-list restricts the display to information about the specified users.

Options

h (no header) Suppresses the header line.

Discussion

The first line that w displays is the same as that displayed by uptime. This line includes the time of day, how long the system has been running (in days, hours, and minutes), how many users are logged in, and how busy the system is (load average). From left to right, the load averages indicate the number of processes that have been waiting to run in the past 1 minute, 5 minutes, and 15 minutes.

The columns of information that w displays have the following headings:

USER TTY FROM LOGIN@ IDLE WHAT

USER is the username of the user. TTY is the device name for the line that the user is on. FROM is the system name that a remote user is logged in from; it is a hyphen for a local user. LOGIN@ gives the date and time the user logged in. IDLE indicates how many minutes have elapsed since the user last used the keyboard. WHAT is the command that user is running.

Examples

The following example shows the full list produced by the w utility:

$ w
20:42  up 2 days, 3:58, 4 users, load averages: 0.04 0.02 0.00
USER    TTY      FROM              LOGIN@  IDLE WHAT
zach    console  -                Sun16   2days -
zach    p1       -                Sun16   26:47 /usr/bin/less -is
zach    p2       -                Mon16       1 vim
zach    p3       bravo.example.co Mon16       - w

wc: Displays the number of lines, words, and bytes

wc [options] [file-list]

The wc utility displays the number of lines, words, and bytes in its input. When you specify more than one file on the command line, wc displays the totals for each file as well as the totals for the group of files.

Arguments

The file-list specifies the pathnames of one or more files that wc analyzes. If you do not specify a file-list, wc takes its input from standard input.

Options

c | (characters) Displays only the number of (8-bit) bytes in the input.
l | (lines, lowercase "l") Displays only the number of lines (that is, NEWLINE characters) in the input.
m | (multibyte) Displays only the number of characters in the input.
w | (words) Displays only the number of words in the input.

Notes

A word is a sequence of characters bounded by SPACEs, TABs, NEWLINEs, or a combination of these characters.

Examples

The following command analyzes the file named memo. The numbers in the output indicate the number of lines, words, and bytes in the file:

$ wc memo
      5     31     146 memo

The next command displays the number of lines and words in three files. The line at the bottom, with the word total in the right column, contains the sum of each column.

$ wc -lw memo1 memo2 memo3
     10     62 memo1
     12     74 memo2
     12     68 memo3
     34    204 total

which: Shows where in PATH a command is located

which command-list

The which utility displays the absolute pathname of commands.

Arguments

The command-list specifies one or more commands (utilities) that which searches for. For each command, which searches the directories listed in the PATH environment variable (page 285), in order, and displays the absolute pathname of the first command (executable file) it finds. If which does not locate a command, it displays an error message.

Options

The which utility does not take any options.

Notes

The TC Shell includes a which builtin (see the tcsh man page) that works slightly differently from the which utility (/usr/bin/which). The which utility does not locate functions or shell builtins because these do not appear in PATH. In contrast, the tcsh which builtin locates aliases, functions, and shell builtins.

The which utility is a csh script and uses the csh startup file (.cshrc) regardless of which shell you call it from.

Examples

The first example quotes the first letter of the command (\which) to prevent the shell from invoking an alias (page 312) for which:

$ \which vim which xxx
/usr/bin/vim
/usr/bin/which
no xxx in /bin /sbin /usr/bin /usr/sbin .

The next example is the same as the previous one except that it is run from tcsh. The tcsh which builtin is used instead of the which utility:

tcsh $ which vim which xxx
/usr/bin/vim
which: shell built-in command.
xxx: Command not found.

who: Displays information about logged-in users

who [options]

who am i

The who utility displays information about users who are logged in on the local system. This information includes each user's username, terminal device, and login time, and, if applicable, the corresponding remote hostname or X display.

Arguments

When given the arguments am i, who displays information about the user giving the command.

Options

Accepts the common options described on page 663.

a | (all) Displays a lot of information.
b | (boot) Displays the date and time the system was last booted.
H | (header) Displays a header.
l | (logins, lowercase "l") Lists devices waiting for a user to log in.
q | (quick) Lists usernames only, followed by the number of users logged in on the system.
T | (tty mode) Appends after each username a character indicating whether that user has messages enabled. A plus (+) means that messages are enabled, a hyphen (-) means that they are disabled, and a question mark (?) indicates that who cannot find the device. If messages are enabled, you can use write (page 63) to communicate with the user. Refer to "mesg: Denies or Accepts Messages" on page 64.
u | (user idle) Includes each user's idle time in the display. If the user has typed on her terminal in the past minute, a period (.) appears in this field. If no input has occurred for more than a day, the word old appears. In addition, this option includes the PID number and comment fields. See "Discussion."

Discussion

The line that who displays has the following syntax:

user [messages] line login-time [idle] comment

The user is the username of the user. The messages indicates whether messages are enabled or disabled (see the T option). The line is the device name associated with the line the user is logged in on. The login-time is the date and time that the user logged in. The idle is the length of time since the terminal was last used (the idle time; see the u option). The comment column is the remote system or X display that the user is logged in from (blank for local users).

Notes

The finger utility (page 734) provides information similar to that given by who.

Examples

The following examples demonstrate the use of the who utility:

$ who
zach     console  May  8 16:46
zach     ttyp1    May  8 16:47
zach     ttyp2    May  9 16:11
zach     ttyp3    May  9 16:15 (bravo.example.co)
$ who am i
zach     ttyp3    May  9 16:15 (bravo.example.co)
$ who -H -u -T
NAME     S LINE     TIME         IDLE  FROM
zach     - console  May  8 16:46  old
zach     + ttyp1    May  8 16:47 00:02
zach     + ttyp2    May  9 16:11   .
zach     + ttyp3    May  9 16:15   .  (bravo.example.co)

xargs: Converts standard input into command lines

xargs [options] [command]

The xargs utility is a convenient, efficient way to convert standard output of one command into arguments for another command. This utility reads from standard input, keeps track of the maximum allowable length of a command line, and avoids exceeding that limit by repeating command as needed. It then executes the newly constructed command line.

Arguments

The command is the command line that xargs uses as a base for the command it constructs. If you do not specify a command, xargs uses echo. The xargs utility appends to command the arguments it receives from standard input. If any arguments should precede the arguments from standard input, you must include them as part of command.

Options

0 | (null, number 0) Arguments from standard input are separated by null characters instead of SPACEs or NEWLINEs. See the print0 action criterion for find (page 730).
I marker
 | (insert) Allows you to place arguments from standard input anywhere within command. A maximum of five occurrences of marker in command for xargs are replaced by the arguments generated from standard input of xargs. With this option, command is executed for each input line.
L [num]
 | (lines) Executes command once for every num lines of input (num defaults to 1).
n num
 | (number) Executes command once for every num arguments in the input line.
p | (prompt) Asks the user for confirmation of each command before running it.
R | replacements
 | (replacements) Specifies the number of times to replace marker with arguments from standard input; the default is 5. Works only with the I option.

Discussion

The xargs utility reads arguments to command from standard input, interpreting each whitespace-delimited string as a separate argument. It then constructs a command line from command and a series of arguments. When the maximum command line length would be exceeded by adding another argument, xargs runs the command line it has built. If more input is present, xargs repeats the process of building a command line and running it. This process continues until all input has been read.

Notes

The xargs utility is often employed as an efficient alternative to the exec option of find (page 729). If you call find with the exec option to run a command, it runs the command once for each file it processes. Each execution of the command creates a new process, which can drain system resources when you are processing many files. By accumulating as many arguments as possible, xargs can greatly reduce the number of processes needed. The first example in the "Examples" section shows how to use xargs with find.

Examples

To locate and remove all files whose names end in .o from the working directory and its subdirectories, you can use find with the exec option:

$ find . -name \* .o -exec rm -f {} \;

This approach calls the rm utility once for each .o file that find locates. Each invocation of rm requires a new process. If a lot of .o files exist, a significant amount of time is spent creating, starting, and then cleaning up these processes. You can reduce the number of processes by allowing xargs to accumulate as many filenames as possible before calling rm:

$ find . -name \* .o -print | xargs rm -f

In the next example, the contents of all * .txt files located by find are searched for lines containing the word login. All filenames that contain login are displayed by grep.

$ find . -name \* .txt -print | xargs grep -w -l login

The next example shows how you can use the I option to cause xargs to embed standard input within command instead of appending it to command. This option also causes command to be executed each time a NEWLINE character is encountered in standard input.

$ cat names
Tom,
Dick,
and Harry
$ xargs echo "Hello, " < names
Hello, Tom, Dick, and Harry
$ xargs -I X echo "Hello X.  Join me for lunch?" <names
Hello Tom,. Join me for lunch?
Hello Dick,. Join me for lunch?
Hello and Harry. Join me for lunch?

The next example uses the same input file as the previous example as well as the n option:

$ xargs echo "Hi there" < names
Hi there Tom, Dick, and Harry
$ xargs -n 1 echo "Hi there" < names
Hi there Tom,
Hi there Dick,
Hi there and
Hi there Harry

The final example shows the use of the print0 criterion to find, and the 0 option to xargs:

$ ls
$ cp /dev/null 'Newline
> File'
$ ls
Newline?File
$ find . -print | xargs ls
ls: ./Newline: No such file or directory
ls: File: No such file or directory
.:
Newline?File
$ find . -print0 | xargs -0 ls
./Newline?File

.:
Newline?File
$

In this example, the potential problem seems harmless. However, systems have been compromised through the use of files with names containing NEWLINEs in conjunction with carelessly written system administration scripts.

See page 849 for another example of the use of xargs.


Part VII: Appendixes

Appendix A REGULAR EXPRESSIONS

Appendix B HELP

Appendix C MAC OS X FOR UNIX USERS

GLOSSARY


Appendix A. Regular Expressions

Previous Page
Next Page

A. Regular Expressions

IN THIS APPENDIX

Characters | 898
Delimiters | 898
Simple Strings | 898
Special Characters | 898
Rules | 901
Bracketing Expressions | 902
The Replacement String | 902
Extended Regular Expressions | 903

A regular expression defines a set of one or more strings of characters. A simple string of characters is a regular expression that defines one string of characters: itself. A more complex regular expression uses letters, numbers, and special characters to define many different strings of characters. A regular expression is said to match any string it defines.

This appendix describes the regular expressions used by ed, vim, emacs, grep, awk, sed, and other utilities. The regular expressions used in shell ambiguous file references are different and are described in "Filename Generation/Pathname Expansion" on page 133.


Characters

As used in this appendix, a character is any character except a NEWLINE. Most characters represent themselves within a regular expression. A special character is one that does not represent itself. If you need to use a special character to represent itself, you must quote it as explained on page 900.


Delimiters

A character called a delimiter usually marks the beginning and end of a regular expression. The delimiter is always a special character for the regular expression it delimits (that is, it does not represent itself but marks the beginning and end of the expression). Although vim permits the use of other characters as a delimiter and grep does not use delimiters at all, the regular expressions in this appendix use a forward slash (/) as a delimiter. In some unambiguous cases, the second delimiter is not required. For example, you can sometimes omit the second delimiter when it would be followed immediately by RETURN.


Simple Strings

The most basic regular expression is a simple string that contains no special characters except the delimiters. A simple string matches only itself (Table A-1). In the examples in this appendix, the strings that are matched are underlined and look like this.

Table A-1. Simple strings

Regular expression

Matches

Examples

/ring/

ring

ring, spring, ringing, stringing

/Thursday/

Thursday

Thursday, Thursday's

/or not/

or not

or not, poor nothing


Special Characters

You can use special characters within a regular expression to cause the regular expression to match more than one string. A regular expression that includes a special character always matches the longest possible string, starting as far toward the beginning (left) of the line as possible.

Periods

A period (.) matches any character (Table A-2).

Table A-2. Period

Regular expression

Matches

Examples

/ .alk/

All strings consisting of a SPACE followed by any character followed by alk

will talk, may balk

/.ing/

All strings consisting of any character preceding ing

sing song, ping, before inglenook

Brackets

Brackets ([]) define a character class[1] that matches any single character within the brackets (Table A-3). If the first character following the left bracket is a caret (^), the brackets define a character class that matches any single character not within the brackets. You can use a hyphen to indicate a range of characters. Within a character-class definition, backslashes and asterisks (described in the following sections) lose their special meanings. A right bracket (appearing as a member of the character class) can appear only as the first character following the left bracket. A caret is special only if it is the first character following the left bracket. A dollar sign is special only if it is followed immediately by the right bracket.

[1] GNU documentation calls these List Operators and defines Character Class operators as expressions that match a predefined group of characters, such as all numbers (see Table VI-30 on page 879).

Table A-3. Brackets

Regular expression

Matches

Examples

/[bB]ill/

Member of the character class b and B followed by ill

bill, Bill, billed

/t[aeiou].k/

t followed by a lowercase vowel, any character, and a k

talkative, stink, teak, tanker

/# [69]/

# followed by a SPACE and a member of the character class 6 through 9

# 60, # 8:, get # 9

/[^azAZ]/

Any character that is not a letter (ASCII character set only)

1, 7, @, ., }, Stop!

Asterisks

An asterisk can follow a regular expression that represents a single character (Table A-4). The asterisk represents zero or more occurrences of a match of the regular expression. An asterisk following a period matches any string of characters. (A period matches any character, and an asterisk matches zero or more occurrences of the preceding regular expression.) A character-class definition followed by an asterisk matches any string of characters that are members of the character class.

Table A-4. Asterisks

Regular expression

Matches

Examples

/ab*c/

a followed by zero or more b's followed by a c

ac, abc, abbc, debbcaabbbc

/ab.*c/

ab followed by zero or more characters followed by c

abc, abxc, ab45c, xab 756.345 x cat

/t.*ing/

t followed by zero or more characters followed by ing

thing, ting, I thought of going

/[azAZ]*/

A string composed only of letters and SPACEs

1. any string without numbers or punctuation!

/(.*)/

As long a string as possible between (and)

Get (this) and (that);

/([^)]*)/

The shortest string possible that starts with (and ends with)

(this), Get (this and that)

Carets and Dollar Signs

A regular expression that begins with a caret (^) can match a string only at the beginning of a line. In a similar manner, a dollar sign ($) at the end of a regular expression matches the end of a line. The caret and dollar sign are called anchors because they force (anchor) a match to the beginning or end of a line (Table A-5).

Table A-5. Carets and dollar signs

Regular expression

Matches

Examples

/^T/

A T at the beginning of a line

This line...,
That Time...,
In Time

/^+[09]/

A plus sign followed by a digit at the beginning of a line

+5 +45.72, +759 Keep this...

/:$/

A colon that ends a line

...below:

Quoting Special Characters

You can quote any special character (but not a digit or a parenthesis) by preceding it with a backslash (Table A-6). Quoting a special character makes it represent itself.

Table A-6. Quoted special characters

Regular expression

Matches

Examples

/end\./

All strings that contain end followed by a period

The end., send., pretend.mail

/\\/

A single backslash

\

/\*/

An asterisk

*.c, an asterisk (*)

/\[5\]/

[5]

it was five [5]

/and\/or/

and/or

and/or


Rules

The following rules govern the application of regular expressions.

Longest Match Possible

A regular expression always matches the longest possible string, starting as far toward the beginning of the line as possible. For example, given the string

This (rug) is not what it once was (a long time ago), is it?

the expression /Th.*is/ matches

This (rug) is not what it once was (a long time ago), is

and /(.*)/ matches

(rug) is not what it once was (a long time ago)

However, /([^)]*)/ matches

(rug)

Given the string

singing songs, singing more and more

the expression /s.*ing/ matches

singing songs, singing

and /s.*ing song/ matches

singing song

Empty Regular Expressions

Within some utilities, such as vim and less (but not grep), an empty regular expression represents the last regular expression that you used. For example, suppose you give vim the following Substitute command:

:s/mike/robert/

If you then want to make the same substitution again, you can use the following command:

:s//robert/

Alternatively, you can use the following commands to search for the string mike and then make the substitution

/mike/
:s//robert/

The empty regular expression (//) represents the last regular expression you used (/mike/).


Bracketing Expressions

You can use quoted parentheses, \( and \), to bracket a regular expression. The string that the bracketed regular expression matches can be recalled, as explained in "Quoted Digit." A regular expression does not attempt to match quoted parentheses. Thus a regular expression enclosed within quoted parentheses matches what the same regular expression without the parentheses would match. The expression /\(rexp\)/ matches what /rexp/ would match; /a\(b*\)c/ matches what /ab*c/ would match.

You can nest quoted parentheses. The bracketed expressions are identified only by the opening \(, so no ambiguity arises in identifying them. The expression /\([az]\([AZ]*\)x\)/ consists of two bracketed expressions, one nested within the other. In the string 3 t dMNORx7 l u, the preceding regular expression matches dMNORx, with the first bracketed expression matching dMNORx and the second matching MNOR.


The Replacement String

The vim and sed editors use regular expressions as search strings within Substitute commands. You can use the ampersand (&) and quoted digits (\n) special characters to represent the matched strings within the corresponding replacement string.

Ampersand

Within a replacement string, an ampersand (&) takes on the value of the string that the search string (regular expression) matched. For example, the following vim Substitute command surrounds a string of one or more digits with NN. The ampersand in the replacement string matches whatever string of digits the regular expression (search string) matched:

:s/[0-9][0-9]*/NN&NN/

Two character-class definitions are required because the regular expression [09]* matches zero or more occurrences of a digit, and any character string constitutes zero or more occurrences of a digit.

Quoted Digit

Within the search string, a bracketed regular expression, \(xxx\), matches what the regular expression would have matched without the quoted parentheses, xxx. Within the replacement string, a quoted digit, \n, represents the string that the bracketed regular expression (portion of the search string) beginning with the nth \( matched. For example, you can take a list of people in the form

last-name, first-name initial

and put it in the form

first-name initial last-name

with the following vim command:

:1,$s/\([^,]*\), \(.*\)/\2 \1/

This command addresses all the lines in the file (1,$). The Substitute command (s) uses a search string and a replacement string delimited by forward slashes. The first bracketed regular expression within the search string, \([^,]*\), matches what the same unbracketed regular expression, [^,]*, would match: zero or more characters not containing a comma (the last-name). Following the first bracketed regular expression are a comma and a SPACE that match themselves. The second bracketed expression, \(.*\), matches any string of characters (the first-name and initial).

The replacement string consists of what the second bracketed regular expression matched (\2), followed by a SPACE and what the first bracketed regular expression matched (\1).


Extended Regular Expressions

The egrep utility, as well as grep and sed when run with the E option, provide all the special characters that are included in ordinary regular expressions, except for \( and \), as well as several others. The vim editor includes the additional characters as well as \( and \). Patterns using the extended set of special characters are called full regular expressions or extended regular expressions.

Two of the additional special characters are the plus sign (+) and the question mark (?). They are similar to *, which matches zero or more occurrences of the previous character. The plus sign matches one or more occurrences of the previous character, whereas the question mark matches zero or one occurrence. You can use any one of the special characters *, +, and ? following parentheses, causing the special character to apply to the string surrounded by the parentheses. Unlike the parentheses in bracketed regular expressions, these parentheses are not quoted (Table A-7).

Table A-7. Extended regular expressions

Regular expression

Matches

Examples

/ab+c/

a followed by one or more b's followed by a c

yabcw, abbc57

/ab?c/

a followed by zero or one b followed by c

back, abcdef

/(ab)+c/

One or more occurrences of the string ab followed by c

zabcd, ababc!

/(ab)?c/

Zero or one occurrence of the string ab followed by c

xc, abcc

In full regular expressions, the vertical bar (|) special character is a Boolean OR operator. Within vim, you must quote the vertical bar by preceding it with a backslash to make it special (\|). A vertical bar between two regular expressions causes a match with strings that match the first expression, the second expression, or both. You can use the vertical bar with parentheses to separate from the rest of the regular expression the two expressions that are being ORed (Table A-8).

Table A-8. Full regular expressions

Regular expression

Meaning

Examples

/ab|ac/

Either ab or ac

ab, ac, abac (abac is two matches of the regular expression)

/^Exit|^Quit/

Lines that begin with Exit or Quit

Exit, Quit, No Exit

/(D|N)\. Jones/

D. Jones or N. Jones

P.D. Jones, N. Jones


Appendix Summary

A regular expression defines a set of one or more strings of characters. A regular expression is said to match any string it defines.

In a regular expression, a special character is one that does not represent itself. Table A-9 lists special characters.

Table A-9. Special characters

Character

Meaning

.

Matches any single character

*

Matches zero or more occurrences of a match of the preceding character

^

Forces a match to the beginning of a line

$

A match to the end of a line

\

Used to quote special characters

\<

Forces a match to the beginning of a word

\>

Forces a match to the end of a word

Table A-10 lists ways of representing character classes and bracketed regular expressions.

Table A-10. Character classes and bracketed regular expressions

Class

Defines

[xyz]

Defines a character class that matches x, y, or z

[^xyz]

Defines a character class that matches any character except x, y, or z

[x-z]

Defines a character class that matches any character x through z inclusive

\(xyz\)

Matches what xyz matches (a bracketed regular expression)

In addition to the preceding special characters and strings (excluding quoted parentheses, except in vim), the characters given in Table A-11 are special within full, or extended, regular expressions.

Table A-11. Extended regular expressions

Expression

Matches

+

Matches one or more occurrences of the preceding character

?

Matches zero or one occurrence of the preceding character

(xyz)+

One or more occurrences of what xyz matches

(xyz)?

Zero or one occurrence of what xyz matches

(xyz)*

Zero or more occurrences of what xyz matches

xyz|abc

Either what xyz or what abc matches (use \| in vim)

(xy|ab)c

Either what xyc or what abc matches (use \| in vim)

Table A-12 lists characters that are special within a replacement string in sed and vim.

Table A-12. Replacement strings

String

Represents

&

Represents what the regular expression (search string) matched

\n

A quoted number, n, represents what the nth bracketed regular expression in the search string matched


Appendix B. Help

Previous Page
Next Page

B. Help

IN THIS APPENDIX

  • Solving a Problem 908

  • Finding Related Information 909

  • Documentation 909

  • Useful UNIX Sites 910

  • Mac OS X Newsgroups and Forums 913

  • Mailing Lists 911

  • Words 911

  • Software 912

  • Specifying a Terminal 913

You need not act as a user or system administrator in isolation. A large community of Mac OS X experts is willing to assist you in learning about, helping you solve your problems with, and getting the most out of OS X. Before you ask for help, however, make sure you have done everything you can to solve the problem by yourself. No doubt someone has experienced the same problem before you and the answer to your question can be found somewhere on the Internet. Your job is to find it. This appendix lists resources and describes methods that can help you in that task.


Solving a Problem

Following is a series of steps that can help you solve a problem without asking someone else for help. Depending on your understanding of and experience with the hardware and software involved, these steps may lead to a solution.

1.
Mac OS X comes with extensive documentation. Read the documentation on the specific hardware or software you are having a problem with. If it is a GNU product, use info; otherwise, use man to find local information. For more information refer to "Getting the Facts: Where to Find Documentation" on page 27. See Table B-1 on page 909 for other sources of documentation.

2.
When the problem involves some type of error or other message, use a search engine, such as Google (www.google.com) or Google Groups (groups.google.com), to look up the message on the Internet. If the message is long, pick a unique part of the message to search for; 10 to 20 characters should be enough. Enclose the search string within double quotation marks.

3.
Use Google (or another search engine) or Google Groups to search on keywords that relate directly to the product and your problem. Read the FAQs.

4.
Check whether the Linux Documentation Project (www.tldp.org) has a HOWTO or mini-HOWTO on the subject in question. Material from the Linux Documentation Project does not always apply to Mac OS X, however.

5.
When all else fails (or perhaps before you try anything else), examine the system logs in /var/log. Running as Superuser or using sudo, first look at the end of the system.log file using the following command:

# tail -20 /var/log/system.log

If system.log contains nothing useful, run the following command. It displays the names of the log files in chronological order, with the most recently modified files appearing at the bottom of the list:

$ ls -ltr /var/log

If your problem involves a network connection, review the secure.log log file on the local and remote systems. Also look at system.log on the remote system. Some systems use different names for these files, most often messages and secure.

6.
The /var/spool directory contains subdirectories with useful information: cups holds the print queues and postfix holds mail server files. Other subdirectories of /var contain similarly useful files.

If you are unable to solve a problem yourself, a thoughtful question to an appropriate newsgroup (page 911) or mailing list (page 911) can elicit useful information. When you send or post a question, make sure you describe the problem and identify the local system carefully. Include the version numbers of the operating system and any software packages that relate to the problem. Also describe your hardware, if appropriate.

The author's home page (www.sobell.com) contains corrections to this book, answers to selected chapter exercises, and pointers to other UNIX and Macintosh sites.


Finding Related Information

Distributions of Mac OS X come with reference pages stored online. You can read these documents by using the info (page 30) or man (page 28) utilities, with the documentation browser in Xcode, or with the Mac Help application. You can read man and info pages to get more information about specific topics while reading this book or to determine which features are available with Mac OS X. You can search for topics by using apropos (see page 59 or give the command man apropos). The Apple support site (www.apple.com/support) has many useful links.

Documentation

Good books are available on various aspects of using and administrating UNIX systems in general. In addition, you may find the sites listed in Table B-1 useful.[1]

[1] The right-hand columns of most of the tables in this appendix show Internet addresses (URLs). All sites have an implicit http:// prefix unless ftp:// or https:// is shown. Refer to "URL: Uniform Resource Locator" on page 421.

Table B-1. Documentation

Site

About the site

URL

Apple Manuals

Apple manuals (look under Software Manuals/Mac OS X in a box on the right).

www.apple.com/support/manuals

GNU Manuals

GNU manuals.

www.gnu.org/manual

Internet FAQ Archives

Searchable FAQ archives.

www.faqs.org

Info

Instructions for using the info utility.

www.gnu.org/software/texinfo/manual/info

RFCs

Request for Comments; see RFC (page 951).

www.rfc-editor.org

The Linux Documentation Project

All things related to Linux documentation (in many languages): HOWTOs, guides, FAQs, man pages, and magazines. This is the best overall source for Linux documentation. Make sure to visit its Links page.

www.tldp.org

Useful UNIX Sites

Sometimes the sites listed in Table B-2 are so busy that you cannot log in. When this happens, you are usually given a list of alternative, or mirror, sites to try.

Table B-2. Useful Mac OS X sites

Site

About the site

URL

Apple Developer Connection

A site for developers; includes information on Intel-based Macintosh systems.

developer.apple.com

Darwin

The Darwin (OS X UNIX) home page.

opendarwin.org developer.apple.com/darwin

GNU

GNU Project Web server.

www.gnu.org

ibiblio

A large library and digital archive. Formerly Metalab; formerly Sunsite.

www.ibiblio.org

MacAddict

A magazine with reviews, tips, and articles about the Macintosh.

www.macaddict.com

MacFixit

A user site full of tips, trivia, and troubleshooting help for the Macintosh.

www.macfixit.com

Mac User Groups

Apple's registry of Macintosh user groups includes a searchable database.

www.apple.com/usergroups/
www.apple.com/usergroups/find/

System Administrators Guild (SAGE)

SAGE is a group for system administrators.

www.sage.org

Sobell

The author's home page contains useful links, errata for this book, code for many of the examples in this book, and answers to selected exercises.

www.sobell.com

Threads

Linux Documentation Project threads FAQ.

tldp.org/FAQ/Threads-FAQ

USENIX

A large, well-established UNIX group. This site has many links, including a list of conferences.

www.usenix.org

VersionTracker

A searchable site covering new releases of software packages.

www.versiontracker.com

X.Org

The X Window System home.

www.x.org

Mac OS X Newsgroups and Forums

One of the best ways to obtain specific information is through a newsgroup. Frequently you can find the answer to your question by reading postings to the newsgroup. Try using Google Groups (groups.google.com) to search through newsgroups to see whether your question has already been asked and answered. Or open a newsreader program and subscribe to appropriate newsgroups. If necessary, you can post your question for someone to answer. Before you do so, make sure you are posting to the correct group and that your question has not been answered already. There is an etiquette to posting questionsfor a good paper written by Eric S. Raymond and Rick Moen, "How To Ask Questions the Smart Way," see www.catb.org/~esr/faqs/smart-questions.html.

The newsgroup comp.sys.mac.system provides postings of solutions to common problems and periodic postings of the most up-to-date versions of the FAQ and similar documents.

Apple Discussions (discussions.info.apple.com) has a list of online forums including a section on Mac OS X and Related Software.

Mailing Lists

Subscribing to a mailing list allows you to participate in an electronic discussion. With most lists, you can send and receive email dedicated to a specific topic to and from a group of users. Moderated lists do not tend to stray as much as unmoderated lists, assuming the list has a good moderator. The disadvantage of a moderated list is that some discussions may be cut off when they get interesting if the moderator deems that the discussion has gone on for too long. Mailing lists described as bulletins are strictly unidirectional: You cannot post information to these lists but can only receive periodic bulletins. If you have the subscription address for a mailing list but are not sure how to subscribe, put the word help in the body and/or header of email that you send to the address. You will usually receive instructions via return email. You can also use a search engine to search for mailing list mac os x.

Apple supports many mailing lists. Visit www.lists.apple.com/mailman/listinfo to find a list of mailing lists; click on the name of a list to display relevant subscription information.

Words

Many dictionaries, thesauruses, and glossaries are available online. Table B-3 lists a few of them.

Table B-3. Looking up words

Site

About the site

URL

ARTFL Project: ROGET'S Thesaurus

Thesaurus

humanities.uchicago.edu/forms_unrest/ROGET.html

DICT.org

Multiple database search for words

www.dict.org

Dictionary.com

Everything related to words

www.dictionary.com

DNS Glossary

DNS glossary

www.menandmice.com/online_docs_and_faq/glossary/glossarytoc.htm

FOLDOC

Computer terms (Free On-Line Dictionary of Computing)

www.foldoc.org

Merriam-Webster

English language

www.m-w.com

OneLook

Multiple-site word search with a single query

www.onelook.com

The Jargon File

An online version of The New Hacker's Dictionary

www.catb.org/~esr/jargon

Webopedia

Commercial technical dictionary

www.webopedia.com

Wikipedia

An open-source (user-contributed) encyclopedia project

wikipedia.org

Wordsmyth

Dictionary and thesaurus

www.wordsmyth.net

Yahoo Reference

Search multiple sources at the same time

education.yahoo.com/reference

Software

There are many ways to learn about interesting software packages and their availability on the Internet. Table B-4 lists sites that you can download software from. Another way to learn about software packages is through a newsgroup (page 911).

Table B-4. Software

Site

About the site

URL

afp548

Long-time distributor of essays and tools

www.afp548.com

BitTorrent

BitTorrent efficiently distributes large amounts of static data

www.bittorrent.com

bombich.com

Home of Carbon Copy Center and other useful tools

bombich.com

CVL

Concurrent Versions Librarian, a GUI for CVS

www.sente.ch/software/cvl

CVS

Concurrent Versions System (see also CVL and TkCVS)

www.nongnu.org/cvs developer.apple.com/internet/opensource/cvsoverview.html

ddd

The ddd utility is a graphical front end for command line debuggers such as gdb

www.gnu.org/software/ddd

fink

The fink utility builds and installs software automatically

fink.sourceforge.net

Free Software Directory

Categorized, searchable lists of free software

www.gnu.org/directory savannah.gnu.org

Freshmeat

A large index of OS X software

osx.freshmeat.net

gcc

Home page for the GNU C compiler

www.gnu.org/software/gcc/gcc.html

gdb

The gdb utility is a command line debugger

www.gnu.org/software/gdb

IceWALKERS

Categorized, searchable lists of free software

www.icewalkers.com

Network Calculators

Subnet mask calculator

www.subnetmask.info

SourceForge

A development Web site with a large repository of open-source code and applications

sourceforge.net

splint

C program verifier (replaces lint)

www.gnu.org/software/gcc/gcc.html

TkCVS

A CVS GUI

www.twobarleycorns.net/tkcvs.html

versiontracker

A large index of OS X software

www.versiontracker.com/osx


Specifying a Terminal

Because vim, emacs, Terminal, and other programs take advantage of features that are specific to various kinds of terminals and terminal emulators, you must tell these programs the name of the terminal you are using or the terminal that your terminal emulator is emulating. Frequently your terminal name is set for you. If your terminal name is not specified or is not specified correctly, your screen will look garbled or, when you start a program, the program will ask what type of terminal you are using.

Terminal names describe the functional characteristics of a terminal or terminal emulator to programs that require this information. Although terminal names are referred to as being either Terminfo or Termcap names, the difference relates to the method that each system uses to store the terminal characteristics internallynot to the manner that you specify the name of a terminal. Terminal names that are often used with terminal emulators and with graphical monitors while they are run in text mode include ansi, vt100, vt102, vt220, and xterm.

When you are running a terminal emulator, you can specify the type of terminal you want to emulate. Set the emulator to either vt100 or vt220, and set TERM to the same value.

When you log in, you may be prompted to identify the type of terminal you are using:

TERM = (vt100)

There are two ways to respond to this prompt. You can press RETURN to set your terminal type to the name in parentheses. Alternatively, when that name does not describe the terminal you are using, you can enter the correct name and then press RETURN.

TERM = (vt100) ansi

You may also receive the following prompt:

TERM = (unknown)

This prompt indicates that the system does not know what type of terminal you are using. If you plan to run programs that require this information, enter the name of the terminal or terminal emulator you are using before you press RETURN.

TERM

If you do not receive a prompt, you can give the following command to display the value of the TERM variable and check whether your terminal type has been set:

$ echo $TERM

If the system responds with the wrong name, a blank line, or an error message, set or change the terminal name. From the Bourne Again Shell (bash), enter a command similar to the following to set the TERM variable so that the system knows the type of terminal you are using:

export TERM=name

Replace name with the terminal name for the terminal you are using, making sure that you do not put a SPACE before or after the equal sign. If you always use the same type of terminal, you can place this command in your ~/.bashrc file (page 259), causing the shell to set the terminal type each time you log in. For example, give the following command to set your terminal name to vt100:

$ export TERM=vt100

Use the following format under the TC Shell (tcsh):

setenv TERM name

Again, replace name with the terminal name for the terminal you are using. Under tcsh you can place this command in your ~/.login file (page 340). For example, under tcsh you can give this command to set your terminal name to vt100:

$ setenv TERM vt100

LANG

For some programs to display information correctly, you may need to set the LANG variable (page 291). Frequently you can set this variable to C. Under bash use the command

$ export LANG=C

Under tcsh use the command

$ setenv LANG C


Appendix C. Mac OS X for UNIX Users

Previous Page
Next Page

C. Mac OS X for UNIX Users

IN THIS APPENDIX

Mac OS X implementation of standard UNIX features 918

This appendix is a brief guide to Mac OS X features and quirks that may be unfamiliar to users who have been using other UNIX or UNIX-like systems. Each entry in Table C-1 describes a difference in behavior and includes cross references to the discussion of the feature in question in the book.

Table C-1. Mac OS X implementation of standard UNIX features

UNIX feature

OS X implementation

/bin/sh

The /bin/sh is a copy of bash (/bin/bash); it is not a link to /bin/bash as it is on some other systems (page 269). The original Bourne Shell does not exist under OS X.

/etc/group

Mac OS X uses NetInfo, not /etc/group, to store group information. For more information see "NetInfo" on page 441 and "Files in /etc are not always used" on page 446.

/etc/passwd

Mac OS X uses NetInfo, not /etc/passwd, to store user accounts. For more information see "NetInfo" on page 441 and "Files in /etc are not always used" endterm="ch11note10.title"/>" on page 446.

ACLs

OS X supports Access Control Lists, which may not be enabled by default (page 97).

Automatic login

By default non-Server Mac OS X systems log in the first user account when you boot the system. For more information refer to "Graphical Login" on page 20.

Case sensitivity

The default OS X filesystem, HFS+, is not case sensitive. For more information refer to "Case sensitivity" on page 75.

Core files

By default OS X does not save core files (page 501). When core files are saved, they are kept in /cores, not in the working directory.

Developer tools

The Developer Tools package, which includes the gcc compiler, is not installed by default (page 478).

Development APIs

Mac OS X uses three software development APIs: Cocoa, Carbon, and BSD UNIX (page 478).

Dynamic linker Id.so

The OS X dynamic linker is dyld, not Id.so (page 488).

ELF and a.out binary formats

The primary binary format under OS X is Mach-O, not ELF or a.out (page 485).

File contents

Files may have multiple forks, which some traditional utilitiesespecially under OS X 10.3 and earliermay corrupt or lose when manipulating files. For more information refer to "Extended Attributes" on page 93.

File metadata

OS X stores more data about a file than traditional UNIX systems store. In addition to traditional UNIX permissions (page 87), OS X files have file flags (page 92) and extended attributes (page 93), which comprise resource forks (page 93), file attributes (page 95), and ACLs (page 97).

File permissions

See "File metadata."

Filesystem structure /etc/fstab

Instead of filesystems being mounted according to settings in /etc/fstab, filesystems are automatically mounted in the /Volumes directory (page 86).

init

Under OS X 10.4 and later, init is replaced by launchd. For more information refer to "The Superserver" on page 456.

LD_LIBRARY_PATH

The variable used to control the dynamic linker is DYLD_LIBRARY_PATH, not LD_LIBRARY_PATH (page 488).

META key

On an Apple keyboard, the META key is labeled OPTION, not ALT or META. For more information see the "Activating the META key" tip on page 31.

Permissions, file

See "File metadata."

root account

Initially, on a non-Server Mac OS X system, there is no password for the root account, so you cannot log in as root and cannot su to root. As a member of the admin group, you can use sudo to run a command or shell with root privileges (page 431). See page 430 for instructions on how to enable the root account.

Runlevels

Mac OS X has only single-user and multiuser modes, not the multiple runlevels of System V UNIX and Linux (page 436).

System databases

Some system databases, such as passwd and group, are stored by NetInfo (page 441) and not in the /etc directory (page 446). You can work with NetInfo databases using the nidump utility (page 796) and the NetInfo Manager GUI application.

vi editor

When you call the vi editor, OS X 10.3 and later run vim (page 145) because the file /usr/bin/vi is a link to /usr/bin/vim. OS X 10.4 and later run vim in compatible mode (page 153). OS X 10.2 and earlier use nvi (page 145) instead of vi.

X Window System

The X Window System is not the primary windowing environment under OS X. If you do run the X Window System, it runs under the native Aqua graphical interface, not as a stand-alone server (page 10).

xinetd

Under OS X version 10.4 and later, xinetd is replaced by launchd. For more information refer to "The Superserver" on page 456.


Glossary

All entries marked with FOLDOC are based on definitions in the Free Online Dictionary of Computing (www.foldoc.org), Denis Howe, editor. Used with permission.

10.0.0.0

See private address space on page 948.

172.16.0.0

See private address space on page 948.

192.168.0.0

See private address space on page 948.

802.11

A family of specifications developed by IEEE for wireless LAN technology, including 802.11 (12 megabits per second), 802.11a (54 megabits per second), 802.11b (11 megabits per second), and 802.11g (54 megabits per second). Apple refers to 802.11b as Airport, and 802.11g as Airport Extreme.

absolute pathname

A pathname that starts with the root directory (/). An absolute pathname locates a file without regard to the working directory.

access

In computer jargon, a verb meaning to use, read from, or write to. To access a file means to read from or write to the file.

Access Control List

See ACL.

access permission

Permission to read from, write to, or execute a file. If you have write access permission to a file, you can write to the file. Also access privilege.

ACL

Access Control List. A system that performs a function similar to file permissions but with much finer-grain control.

active window

On a desktop, the window that receives the characters you type on the keyboard. Same as focus, desktop (page 933).

address mask

See subnet mask on page 956.

Airport

Apple's name for 802.11 wireless technology.

alias

A mechanism of a shell that enables you to define new commands. Also, in Macintosh documentation, a near-synonym for link (page 940).

alphanumeric character

One of the characters, either uppercase or lowercase, from A to Z and 0 to 9, inclusive.

ambiguous file reference

A reference to a file that does not necessarily specify any one file but can be used to specify a group of files. The shell expands an ambiguous file reference into a list of filenames. Special characters represent single characters (?), strings of zero or more characters (*), and character classes ([]) within ambiguous file references. An ambiguous file reference is a type of regular expression (page 950).

angle bracket

A left angle bracket (<) and a right angle bracket (>). The shell uses < to redirect a command's standard input to come from a file and > to redirect the standard output. The shell uses the characters << to signify the start of a Here document and >> to append output to a file.

animate

When referring to a window action, means that the action is slowed down so the user can view it. For example, when you minimize a window, it can disappear all at once (not animated) or it can slowly telescope into the panel so you can get a visual feel for what is happening (animated).

anti-aliasing

Adding gray pixels at the edge of a diagonal line to get rid of the jagged appearance and thereby make the line look smoother. Anti-aliasing sometimes makes type on a screen look better and sometimes worse; it works best on small and large fonts and is less effective on fonts from 8 to 15 points. See also subpixel hinting (page 956).

API

Application program interface. The interface (calling conventions) by which an application program accesses an operating system and other services. An API is defined at the source code level and provides a level of abstraction between the application and the kernel (or other privileged utilities) to ensure the portability of the code.FOLDOC

append

To add something to the end of something else. To append text to a file means to add the text to the end of the file. The shell uses >> to append a command's output to a file.

applet

A small program that runs within a larger program. Examples are Java applets that run in a browser and panel applets that run from a desktop panel.

archive

A file that contains a group of smaller, typically related, files. Also, to create such a file. The tar, cpio, and pax utilities can create and read archives.

argument

A number, letter, filename, or another string that gives some information to a command and is passed to the command when it is called. A command line argument is anything on a command line following the command name that is passed to the command. An option is a kind of argument.

arithmetic expression

A group of numbers, operators, and parentheses that can be evaluated. When you evaluate an arithmetic expression, you end up with a number. The Bourne Again Shell uses the expr command to evaluate arithmetic expressions; the TC Shell uses @; and the Z Shell uses let.

array

An arrangement of elements (numbers or strings of characters) in one or more dimensions. The Bourne Again, TC, and Z Shells and awk can store and process arrays.

ASCII

American Standard Code for Information Interchange. A code that uses seven bits to represent both graphic (letters, numbers, and punctuation) and CONTROL characters. You can represent textual information, including program source code and English text, in ASCII code. Because ASCII is a standard, it is frequently used when exchanging information between computers. See the file /usr/share/misc/ascii or give the command man ascii to see a list of ASCII codes.

Extensions of the ASCII character set use eight bits. The seven-bit set is common; the eight-bit extensions are still coming into popular use. The eighth bit is sometimes referred to as the metabit.

ASCII terminal

A text-based terminal. Contrast with graphical display (page 934).

ASP

Application service provider. A company that provides applications over the Internet.

asynchronous event

An event that does not occur regularly or synchronously with another event. Mac OS X system signals are asynchronous; they can occur at any time because they can be initiated by any number of nonregular events.

attachment

A file that is attached to, but is not part of, a piece of email. Attachments are frequently opened by programs (including your Internet browser) that are called by your mail program so you may not be aware that they are not an integral part of an email message.

attributes

Flags and other data associated with a file. Under OS X, includes file attributes (page 932). See also extended attributes (page 932).

authentication

The verification of the identity of a person or process. In a communication system, authentication verifies that a message comes from its stated source. PAM handles authentication on an OS X system.FOLDOC

automatic mounting

A way of demand mounting directories from remote hosts without having them hard configured into /etc/fstab. Also called automounting. By default Mac OS X automounts remote directories.

avoided

An object, such as a panel, that should not normally be covered by another object, such as a window.

back door

A security hole deliberately left in place by the designers or maintainers of a system. The motivation for creating such holes is not always sinister; some operating systems, for example, come out of the box with privileged accounts intended for use by field service technicians or the vendor's maintenance programmers.

Ken Thompson's 1983 Turing Award lecture to the ACM revealed the existence, in early UNIX versions, of a back door that may be the most fiendishly clever security hack of all time. The C compiler contained code that would recognize when the login command was being recompiled and would insert some code recognizing a password chosen by Thompson, giving him entry to the system whether or not an account had been created for him.

Normally such a back door could be removed by removing it from the source code for the compiler and recompiling the compiler. But to recompile the compiler, you have to use the compiler, so Thompson arranged that the compiler would recognize when it was compiling a version of itself. It would insert into the recompiled compiler the code to insert into the recompiled login the code to allow Thompson entry, and, of course, the code to recognize itself and do the whole thing again the next time around. Having done this once, he was then able to recompile the compiler from the original sources; the hack perpetuated itself invisibly, leaving the back door in place and active but with no trace in the sources.

Sometimes called a wormhole. Also trap door.FOLDOC

background process

A process that is not run in the foreground. Also called a detached process, a background process is initiated by a command line that ends with an ampersand (&). You do not have to wait for a background process to run to completion before giving the shell additional commands. If you have job control, you can move background processes to the foreground, and vice versa.

basename

The name of a file that, in contrast with a pathname, does not mention any of the directories containing the file (and therefore does not contain any slashes [/]). For example, hosts is the basename of /etc/hosts.FOLDOC

baud

The maximum information-carrying capacity of a communication channel in symbols (state transitions or level transitions) per second. It coincides with bits per second only for two-level modulation with no framing or stop bits. A symbol is a unique state of the communication channel, distinguishable by the receiver from all other possible states. For example, it may be one of two voltage levels on a wire for a direct digital connection, or it might be the phase or frequency of a carrier.FOLDOC

Baud is often mistakenly used as a synonym for bits per second.

baud rate

Transmission speed. Usually used to measure terminal or modem speed. Common baud rates range from 110 to 38,400 baud. See baud.

Berkeley UNIX

One of the two major versions of the UNIX operating system. Berkeley UNIX was developed at the University of California at Berkeley by the Computer Systems Research Group and is often referred to as BSD (Berkeley Software Distribution).

BIND

Berkeley Internet Name Domain. An implementation of a DNS (page 930) server developed and distributed by the University of California at Berkeley

BIOS

Basic Input/Output System. On PCs, EEPROM-based (page 931) system software that provides the lowest-level interface to peripheral devices and controls the first stage of the bootstrap (page 923) process, which loads the operating system. The BIOS can be stored in different types of memory. The memory must be nonvolatile so that it remembers the system settings even when the system is turned off. Also BIOS ROM.

bit

The smallest piece of information a computer can handle. A bit is a binary digit: either 1 or 0 (on or off).

bit depth

Same as color depth (page 926).

bit-mapped display

A graphical display device in which each pixel on the screen is controlled by an underlying representation of zeros and ones.

blank character

Either a SPACE or a TAB character, also called whitespace (page 961). In some contexts, NEWLINEs are considered blank characters.

block

A section of a disk or tape (usually 1,024 bytes long but shorter or longer on some systems) that is written at one time.

block device

A disk or tape drive. A block device stores information in blocks of characters. A block device is represented by a block device (block special) file. Contrast with character device (page 925).

block number

Disk and tape blocks are numbered so that Mac OS X can keep track of the data on the device.

blocking factor

The number of logical blocks that make up a physical block on a tape or disk. When you write 1K logical blocks to a tape with a physical block size of 30K, the blocking factor is 30.

boot

See bootstrap.

boot loader

A very small program that takes its place in the bootstrap process that brings a computer from off or reset to a fully functional state.

bootstrap

Derived from "Pull oneself up by one's own bootstraps," the incremental process of loading an operating system kernel into memory and starting it running without any outside assistance. Frequently shortened to boot.

Bourne Again Shell

bash. GNU's command interpreter for UNIX, bash is a POSIX-compliant shell with full Bourne Shell syntax and some C Shell commands built in. The Bourne Again Shell supports emacs-style command line editing, job control, functions, and online help.FOLDOC

Bourne Shell

sh. This UNIX command processor was developed by Steve Bourne at AT&T Bell Laboratories.

brace

A left brace ({) and a right brace (}). Braces have special meanings to the shell.

bracket

A square bracket (page 955) or an angle bracket (page 920).

branch

In a tree structure, a branch connects nodes, leaves, and the root. The Mac OS X filesystem hierarchy is often conceptualized as an upside-down tree. The branches connect files and directories. In a source code control system, such as SCCS or RCS, a branch occurs when a revision is made to a file and is not included in subsequent revisions to the file.

bridge

Typically a two-port device originally used for extending networks at layer 2 (data link) of the Internet Protocol model.

broadcast

A transmission to multiple, unspecified recipients. On Ethernet a broadcast packet is a special type of multicast packet that has a special address indicating that all devices that receive it should process it. Broadcast traffic exists at several layers of the network stack, including Ethernet and IP. Broadcast traffic has one source but indeterminate destinations (all hosts on the local network).

broadcast address

The last address on a subnet (usually 255), reserved as shorthand to mean all hosts.

broadcast network

A type of network, such as Ethernet, in which any system can transmit information at any time, and all systems receive every message.

BSD

See Berkeley UNIX on page 923.

buffer

An area of memory that stores data until it can be used. When you write information to a file on a disk, Mac OS X stores the information in a disk buffer until there is enough to write to the disk or until the disk is ready to receive the information.

bug

An unwanted and unintended program property, especially one that causes the program to malfunction.FOLDOC

builtin (command)

A command that is built into a shell. Each of the three major shellsthe Bourne Again, TC, and Z Shellshas its own set of builtins. Refer to "Builtins" on page 138.

byte

A component in the machine data hierarchy, usually larger than a bit and smaller than a word; now most often eight bits and the smallest addressable unit of storage. A byte typically holds one character.FOLDOC

C programming language

A modern systems language that has high-level features for efficient, modular programming as well as lower-level features that make it suitable for use as a systems programming language. It is machine independent so that carefully written C programs can be easily transported to run on different machines. Most of the Mac OS X operating system is written in C, and Mac OS X provides an ideal environment for programming in C.

C Shell

csh. The C Shell command processor was developed by Bill Joy for BSD UNIX. It was named for the C programming language because its programming constructs are similar to those of C. See shell on page 953.

cable modem

A type of modem that allows you to access the Internet by using your cable television connection.

cache

Holding recently accessed data, a small, fast memory designed to speed up subsequent access to the same data. Most often applied to processor-memory access but also used for a local copy of data accessible over a network, from a hard disk, and so on.FOLDOC

calling environment

A list of variables and their values that is made available to a called program. Refer to "Executing a Command" on page 294.

Carbon

An API (page 921) that has been in use since Mac OS 9 and provides compatibility with pre-UNIX Mac OS systems (OS 9 and later). Contrast with Cocoa (page 926).

cascading stylesheet

See CSS on page 928.

cascading windows

An arrangement of windows such that they overlap, generally with at least part of the title bar visible. Opposite of tiled windows (page 958).

case sensitive

Able to distinguish between uppercase and lowercase characters. Unless you set the ignorecase parameter, vim performs case-sensitive searches. The grep utility performs case-sensitive searches unless you use the i option.

catenate

To join sequentially, or end to end. The Mac OS X cat utility catenates files: It displays them one after the other. Also concatenate.

chain loading

The technique used by a boot loader to load unsupported operating systems. Used for loading such operating systems as DOS or Windows, it works by loading another boot loader.

character-based

A program, utility, or interface that works only with ASCII (page 921) characters. This set of characters includes some simple graphics, such as lines and corners, and can display colored characters. It cannot display true graphics. Contrast with GUI (page 934).

character-based terminal

A terminal that displays only characters and very limited graphics. See character-based.

character class

In a regular expression, a group of characters that defines which characters can occupy a single character position. A character-class definition is usually surrounded by square brackets. The character class defined by [abcr] represents a character position that can be occupied by a, b, c, or r. Also list operator.

In POSIX, used to refer to sets of characters with a common characteristic, denoted by the notation [:class:]; for example, [:upper:] denotes the set of uppercase letters.

character device

A terminal, printer, or modem. A character device stores or displays characters one at a time. A character device is represented by a character device (character special) file. Contrast with block device (page 923).

checksum

A computed value that depends on the contents of a block of data and is transmitted or stored along with the data to detect corruption of the data. The receiving system recomputes the checksum based on the received data and compares this value with the one sent with the data. If the two values are the same, the receiver has some confidence that the data was received correctly.

The checksum may be 8, 16, or 32 bits, or some other size. It is computed by summing the bytes or words of the data block, ignoring overflow. The checksum may be negated so that the total of the data words plus the checksum is zero.

Internet packets use a 32-bit checksum.FOLDOC

child process

A process that is created by another process, the parent process. Every process is a child process except for the first process, which is started when Mac OS X begins execution. When you run a command from the shell, the shell spawns a child process to run the command. See process on page 948.

CIDR

Classless Inter-Domain Routing. A scheme that allocates blocks of Internet addresses in a way that allows summarization into a smaller number of routing table entries. A CIDR block is a block of Internet addresses assigned to an ISP by the Internic.FOLDOC

CIFS

Common Internet File System. An Internet filesystem protocol based on SMB (page 953). CIFS runs on top of TCP/IP, uses DNS, and is optimized to support slower dial-up Internet connections. SMB and CIFS are used interchangeably.FOLDOC

CIPE

Crypto IP Encapsulation (page 931). This protocol (page 949) tunnels (page 959) IP packets within encrypted UDP (page 959) packets, is lightweight and simple, and works over dynamic addresses, NAT (page 944), and SOCKS (page 954) proxies (page 949).

cipher (cypher)

A cryptographic system that uses a key to transpose/substitute characters within a message, the key itself, or the message.

ciphertext

Text that is encrypted. Contrast with plaintext (page 947).

Classless Inter-Domain Routing

See CIDR.

cleartext

Text that is not encrypted; also plaintext. Contrast with ciphertext.

CLI

Command line interface. See also character-based (page 925).

client

A computer or program that requests one or more services from a server.

Cocoa

The native API under Mac OS X. Cocoa is not compatible with Mac OS 9 and earlier. Contrast with Carbon (page 925).

CODEC

Coder/decoder or compressor/decompressor. A hardware and/or software technology that codes and decodes data. MPEG is a popular CODEC for computer video.

color depth

The number of bits used to generate a pixelusually 8, 16, 24, or 32. The color depth is directly related to the number of colors that can be generated. The number of colors that can be generated is 2 raised to the color-depth power. Thus a 24-bit video adapter can generate about 16.7 million colors.

color quality

See color depth.

combo box

A combination of a list and text entry box. A user can either select an option from a provided list or enter his own option.

command

What you give the shell in response to a prompt. When you give the shell a command, it executes a utility, another program, a builtin command, or a shell script. Utilities are often referred to as commands. When you are using an interactive utility, such as vim or mail, you use commands that are appropriate to that utility.

command line

A line containing instructions and arguments that executes a command. This term usually refers to a line that you enter in response to a shell prompt on a character-based terminal or terminal emulator.

command substitution

Replacing a command with its output. The shells perform command substitution when you enclose a command between $( and ) or between a pair of back ticks (''), also called grave accent marks.

component architecture

A notion in object-oriented programming where "components" of a program are completely generic. Instead of having a specialized set of methods and fields, they have generic methods through which the component can advertise the functionality it supports to the system into which it is loaded. This strategy enables completely dynamic loading of objects. JavaBeans is an example of a component architecture.FOLDOC

concatenate

See catenate on page 925.

condition code

See exit status on page 932.

connection-oriented protocol

A type of transport layer data communication service that allows a host to send data in a continuous stream to another host. The transport service guarantees that all data will be delivered to the other end in the same order as sent and without duplication. Communication proceeds through three well-defined phases: connection establishment, data transfer, and connection release. The most common example is TCP (page 957).

Also called connection-based protocol and stream-oriented protocol. Contrast with connectionless protocol and datagram (page 929).FOLDOC

connectionless protocol

The data communication method in which communication occurs between hosts with no previous setup. Packets sent between two hosts may take different routes. There is no guarantee that packets will arrive as transmitted or even that they will arrive at the destination at all. UDP (page 959) is a connectionless protocol. Also called packet switching. Contrast with circuit switching and connection-oriented protocol.FOLDOC

console

See system console on page 957.

console terminal

See system console on page 957.

control character

A character that is not a graphic character, such as a letter, number, or punctuation mark. Such characters are called control characters because they frequently act to control a peripheral device. RETURN and FORMFEED are control characters that control a terminal or printer.

The word CONTROL is shown in this book in THIS FONT because it is a key that appears on most terminal keyboards. Control characters are represented by ASCII codes less than 32 (decimal). See also nonprinting character on page 945.

control structure

A statement used to change the order of execution of commands in a shell script or other program. Each shell provides control structures (for example, if and while) as well as other commands that alter the order of execution (for example, exec). Also control flow commands.

cookie

Data stored on a client system by a server. The client system browser sends the cookie back to the server each time it accesses that server. For example, a catalog shopping service may store a cookie on your system when you place your first order. When you return to the site, it knows who you are and can supply your name and address for subsequent orders. You may consider cookies to be an invasion of privacy.

CPU

Central processing unit. The part of a computer that controls all the other parts. The CPU includes the control unit and the arithmetic and logic unit (ALU). The control unit fetches instructions from memory and decodes them to produce signals that control the other parts of the computer. These signals can cause data to be transferred between memory and ALU or peripherals to perform input or output. A CPU that is housed on a single chip is called a microprocessor. Also processor and central processor.

cracker

An individual who attempts to gain unauthorized access to a computer system. These individuals are often malicious and have many means at their disposal for breaking into a system. Contrast with hacker (page 934).FOLDOC

crash

The system suddenly and unexpectedly stops or fails. Derived from the action of the hard disk heads on the surface of the disk when the air gap between the two collapses.

cryptography

The practice and study of encryption and decryptionencoding data so that only a specific individual or machine can decode it. A system for encrypting and decrypting data is a cryptosystem. Such systems usually rely on an algorithm for combining the original data (plaintext) with one or more keysnumbers or strings of characters known only to the sender and/or recipient. The resulting output is called ciphertext (page 926).

The security of a cryptosystem usually depends on the secrecy of keys rather than on the supposed secrecy of an algorithm. Because a strong cryptosystem has a large range of keys, it is not possible to try all of them. Ciphertext appears random to standard statistical tests and resists known methods for breaking codes.FOLDOC

.cshrc file

In your home directory, a file that the TC Shell executes each time you invoke a new TC Shell. You can use this file to establish variables and aliases.

CSS

Cascading stylesheet. Describes how documents are presented on screen and in print. Attaching a stylesheet to a structured document can affect the way it looks without adding new HTML (or other) tags and without giving up device independence. Also stylesheet.

current (process, line, character, directory, event, and so on)

The item that is immediately available, working, or being used. The current process is the program you are running, the current line or character is the one the cursor is on, and the current directory is the working directory.

cursor

A small lighted rectangle, underscore, or vertical bar that appears on the terminal screen and indicates where the next character will appear. Differs from the mouse pointer (page 943).

daemon

A program that is not invoked explicitly but lies dormant, waiting for some condition(s) to occur. The perpetrator of the condition need not be aware that a daemon is lurking (although often a program will commit an action only because it knows that it will implicitly invoke a daemon). From the mythological meaning, later rationalized as the acronym Disk And Execution MONitor.FOLDOC

data fork

One of two fork (page 933) of a file on an HFS+ filesystem HFS+ (page 935). The data fork holds arbitrary data. Contrast with resource fork (page 951).

data structure

A particular format for storing, organizing, working with, and retrieving data. Frequently, data structures are designed to work with specific algorithms that facilitate these tasks. Common data structures include trees, files, records, tables, arrays, and so on.

datagram

A self-contained, independent entity of data carrying sufficient information to be routed from the source to the destination computer without reliance on earlier exchanges between this source and destination computer and the transporting network. UDP (page 959) uses datagrams; IP (page 938) uses packets (page 946). Packets are indivisible at the network layer; datagrams are not.FOLDOC See also frame (page 933).

dataless

A computer, usually a workstation, that uses a local disk to boot a copy of the operating system and access system files but does not use a local disk to store user files.

dbm

A standard, simple database manager. Implemented as gdbm (GNU database manager), it uses hashes to speed searching. The most common versions of the dbm database are dbm, ndbm, and gdbm.

DDoS attack

Distributed denial of service attack. A DoS attack (page 931) from many systems that do not belong to the perpetrator of the attack.

debug

To correct a program by removing its bugs (that is, errors).

default

Something that is selected without being explicitly specified. For example, when used without an argument, ls displays a list of the files in the working directory by default.

delta

A set of changes made to a file that has been encoded by the Source Code Control System (SCCS).

denial of service

See DoS attack on page 931.

dereference

When speaking of symbolic links, follow the link rather than working with the reference to the link. For example, the L option causes ls to list the entry that a symbolic link points to rather than the symbolic link (the reference) itself.

desktop

A collection of windows, toolbars, icons, and buttons, some or all of which appear on your display. A desktop comprises one or more workspaces (page 961).

desktop manager

An icon- and menu-based user interface to system services that allows you to run applications and use the filesystem without using the system's command line interface.

detached process

See background process on page 922.

device

A disk drive, printer, terminal, plotter, or other input/output unit that can be attached to the computer.

device driver

Part of the Mac OS X kernel that controls a device, such as a terminal, disk drive, or printer.

device file

A file that represents a device. Also special file.

device filename

The pathname of a device file. Mac OS X systems have two kinds of device files: block and character device files. Mac OS X also has FIFOs (named pipes) and sockets. Device files are traditionally located in the /dev directory.

device number

See major device number (page 942) and minor device number (page 943).

DHCP

Dynamic Host Configuration Protocol. A protocol that dynamically allocates IP addresses to computers on a LAN.FOLDOC

directory

Short for directory file. A file that contains a list of other files.

directory hierarchy

A directory, called the root of the directory hierarchy, and all the directory and ordinary files below it (its children).

directory service

A structured repository of information on people and resources within an organization, facilitating management and communication.FOLDOC

disk partition

See partition on page 946.

diskless

A computer, usually a workstation, that has no disk and must contact another computer (a server) to boot a copy of the operating system and access the necessary system files.

distributed computing

A style of computing in which tasks or services are performed by a network of cooperating systems, some of which may be specialized.

DMZ

Demilitarized zone. A host or small network that is a neutral zone between a LAN and the Internet. It can serve Web pages and other data to the Internet and allow local systems access to the Internet while preventing LAN access to unauthorized Internet users. Even if a DMZ is compromised, it holds no data that is private and none that cannot be easily reproduced.

DNS

Domain Name Service. A distributed service that manages the correspondence of full hostnames (those that include a domain name) to IP addresses and other system characteristics.

DNS domain name

See domain name.

document object model

See DOM.

DOM

Document Object Model. A platform-/language-independent interface that enables a program to update the content, structure, and style of a document dynamically. The changes can then be made part of the displayed document. Go to www.w3.org/DOM for more information.

domain name

A name associated with an organization, or part of an organization, to help identify systems uniquely. Technically, the part of the FQDN (page 933) to the right of the leftmost period. Domain names are assigned hierarchically. The domain berkeley.edu refers to the University of California at Berkeley, for example; it is part of the top-level edu (education) domain. Also DNS domain name. Different than NIS domain name (page 945).

Domain Name Service

See DNS.

door

An evolving filesystem-based RPC (page 952) mechanism.

DoS attack

Denial of service attack. An attack that attempts to make the target host or network unusable by flooding it with spurious traffic.

DPMS

Display Power Management Signaling. A standard that can extend the life of CRT monitors and conserve energy. DPMS supports four modes for a monitor: Normal, Standby (power supply on, monitor ready to come to display images almost instantly), Suspend (power supply off, monitor takes up to ten seconds to display an image), and Off.

drag

To move an icon from one position or application to another, usually in the context of a window manager. The motion part of drag-and-drop.

DSA

Digital Signature Algorithm. A public key cipher used to generate digital signatures.

DSL

Digital Subscriber Line/Loop. Provides high-speed digital communication over a specialized, conditioned telephone line. See also xDSL (page 962).

Dynamic Host Configuration Protocol

See DHCP on page 930.

editor

A utility, such as vim or emacs, that creates and modifies text files.

EEPROM

Electrically erasable, programmable, readonly memory. A PROM (page 948) that can be written to.

effective user ID

The user ID that a process appears to have; usually the same as the user ID. For example, while you are running a setuid program, the effective user ID of the process running the program is that of the owner of the program.

element

One thing; usually a basic part of a group of things. An element of a numeric array is one of the numbers stored in the array.

emoticon

See smiley on page 954.

encapsulation

See tunneling on page 959.

environment

See calling environment on page 925.

EOF

End of file.

EPROM

Erasable, programmable, readonly memory. A PROM (page 948) that can be written to by applying a higher than normal voltage.

escape

See quote on page 949.

Ethernet

A type of LAN (page 940) capable of transfer rates as high as 1,000 megabits per second.

event

An occurrence, or happening, of significance to a task or programfor example, the completion of an asynchronous input/output operation, such as a keypress or mouse click.FOLDOC

exabyte

260 bytes or about 1018 bytes. See also large number (page 940).

exit status

The status returned by a process; either successful (usually 0) or unsuccessful (usually 1).

exploit

A security hole or an instance of taking advantage of a security hole.FOLDOC

expression

See logical expression (page 941) and arithmetic expression (page 921).

extended attributes

Additional information associated with a file. Includes file attributes, ACLs (page 920), and the resource fork (page 951).

extranet

A network extension for a subset of users (such as students at a particular school or engineers working for the same company). An extranet limits access to private information even though it travels on the public Internet.

failsafe session

A session that allows you to log in on a minimal desktop in case your standard login does not work well enough to allow you to log in to fix a login problem.

FDDI

Fiber Distributed Data Interface. A type of LAN (page 940) designed to transport data at the rate of 100 million bits per second over fiberoptic cable.

file

A collection of related information referred to with a filename and frequently stored on a disk. Text files typically contain memos, reports, messages, program source code, lists, or manuscripts. Binary or executable files contain utilities or programs that you can run. Refer to "Directory Files and Ordinary Files" on page 73.

file attributes

A file's type and creator codes, timestamps, and attribute flags (page 95). Part of the file's extended attributes.

filename

The name of a file. A filename refers to a file.

filename completion

Automatic completion of a filename after you specify a unique prefix.

filename extension

The part of a filename following a period.

filename generation

What occurs when the shell expands ambiguous file references. See ambiguous file reference on page 920.

filesystem

A data structure (page 929) that usually resides on part of a disk. Mac OS X systems have a root filesystem (startup volume), and many have other filesystems. Each filesystem is composed of some number of blocks, depending on the size of the disk partition that has been assigned to the filesystem. Different filesystems use different structures. Each UFS (page 959) filesystem has a control block, named the superblock, that contains information about the filesystem. The other blocks in a filesystem are inodes, which contain control information about individual files, and data blocks, which contain the information in the files. See also HFS+ (page 935).

filling

A variant of maximizing in which window edges are pushed out as far as they can go without overlapping another window.

filter

A command that can take its input from standard input and send its output to standard output. A filter transforms the input stream of data and sends it to standard output. A pipe usually connects a filter's input to standard output of one command, and a second pipe connects the filter's output to standard input of another command. The grep and sort utilities are commonly used as filters.

firewall

A device for policy-based traffic management used to keep a network secure. A firewall can be implemented in a single router that filters out unwanted packets, or it can rely on a combination of routers, proxy servers, and other devices. Firewalls are widely used to give users access to the Internet in a secure fashion and to separate a company's public WWW server from its internal network. They are also employed to keep internal network segments more secure.

Recently the term has come to be defined more loosely to include a simple packet filter running on an endpoint machine.

See also proxy server on page 949.

firmware

Software built into a computer, often in ROM (page 951). May be used as part of the bootstrap (page 923) procedure.

focus, desktop

On a desktop, the window that is active. The window with the desktop focus receives the characters you type on the keyboard. Same as active window (page 920).

footer

The part of a format that goes at the bottom (or foot) of a page. Contrast with header (page 935).

foreground process

When you run a command in the foreground, the shell waits for the command to finish before giving you another prompt. You must wait for a foreground process to run to completion before you can give the shell another command. If you have job control, you can move background processes to the foreground, and vice versa. See job control on page 939. Contrast with background process (page 922).

fork

1. To create a process. When one process creates another process, it is said to fork a process. Also spawn.

2. One of multiple components of a file sharing a single name, on the HFS+ filesystem (page 935). Specifically, a data fork (page 929) or a resource fork (page 951).

FQDN

Fully qualified domain name. The full name of a system, consisting of its hostname and its domain name, including the top-level domain. Technically the name that gethostbyname(2) returns for the host named by gethostname(2). For example, speedy is a hostname and speedy.example.com is an FQDN. An FQDN is sufficient to determine a unique Internet address for a machine on the Internet.FOLDOC

frame

A data link layer packet that contains, in addition to data, the header and trailer information required by the physical medium. Network layer packets are encapsulated to become frames.FOLDOC See also datagram (page 929) and packet (page 946).

free list

In a filesystem, the list of blocks that are available for use. Information about the free list is kept in the superblock of a UFS (page 959) filesystem.

free space

The portion of a hard disk that is not within a partition. A new hard disk has no partitions and contains all free space.

full duplex

The ability to receive and transmit data simultaneously. A network switch (page 945) is typically a full-duplex device. Contrast with half-duplex (page 934).

fully qualified domain name

See FQDN.

function

See shell function on page 953.

gateway

A generic term for a computer or a special device connected to more than one dissimilar type of network to pass data between them. Unlike a router, a gateway often must convert the information into a different format before passing it on. The historical usage of gateway to designate a router is deprecated.

GCOS

See GECOS.

GECOS

General Electric Comprehensive Operating System. For historical reasons, the user information field in the passwd database is called the GECOS field. Also GCOS.

giga-

In the binary system, the prefix giga- multiplies by 230 (i.e., 1,073,741,824). Gigabit and gigabyte are common uses of this prefix. Abbreviated as G. See also large number on page 940.

glyph

A symbol that communicates a specific piece of information nonverbally. A smiley (page 954) is a glyph.

GMT

Greenwich Mean Time. See UTC on page 960.

graphical display

A bitmapped monitor that can display graphical images. Contrast with ASCII terminal (page 921).

graphical user interface

See GUI.

group (of users)

A collection of users. Groups are used as a basis for determining file access permissions. If you are not the owner of a file and you belong to the group the file is assigned to, you are subject to the group access permissions for the file. A user can simultaneously belong to several groups.

group (of windows)

A way to identify similar windows so they can be displayed and acted on similarly. Typically windows started by a given application belong to the same group.

group ID

A unique number that identifies a set of users. It is stored in the passwd and group databases. The group database associates group IDs with group names.

GUI

Graphical user interface. A GUI provides a way to interact with a computer system by choosing items from menus or manipulating pictures drawn on a display screen instead of by typing command lines. Under Mac OS X, the Aqua environment provides a graphical display and mouse/keyboard input. Contrast with character-based (page 925).

hacker

A person who enjoys exploring the details of programmable systems and learning how to stretch their capabilities, as opposed to users, who prefer to learn only the minimum necessary. One who programs enthusiastically (even obsessively) or who enjoys programming rather than just theorizing about programming.FOLDOC Contrast with cracker (page 928).

half-duplex

A half-duplex device can only receive or transmit at a given moment; it cannot do both. A hub (page 936) is typically a half-duplex device. Contrast with full duplex (page 933).

hard link

A directory entry that contains the filename and inode number for a file. The inode number identifies the location of control information for the file on the disk, which in turn identifies the location of the file's contents on the disk. Every file has at least one hard link, which locates the file in a directory. When you remove the last hard link to a file, you can no longer access the file. See link (page 940) and symbolic link (page 957).

hash

A string that is generated from another string. See one-way hash function on page 945. When used for security, a hash can prove, almost to a certainty, that a message has not been tampered with during transmission: The sender generates a hash of a message, encrypts the message and hash, and sends the encrypted message and hash to the recipient. The recipient decrypts the message and hash, generates a second hash from the message, and compares the hash that the sender generated to the new hash. When they are the same, the message has probably not been tampered with. A hash can also be used to create an index called a hash table. Also hash value.

hash table

An index created from hashes of the items to be indexed. The hash function makes it highly unlikely that two items will create the same hash. To look up an item in the index, create a hash of the item and search for the hash. Because the hash is typically shorter than the item, the search is more efficient.

header

When you are formatting a document, the header goes at the top, or head, of a page. In electronic mail the header identifies who sent the message, when it was sent, what the subject of the message is, and so forth.

Here document

A shell script that takes its input from the file that contains the script.

hesiod

The nameserver of project Athena. Hesiod is a name service library that is derived from BIND (page 923) and leverages a DNS infrastructure.

heterogeneous

Consisting of different parts. A heterogeneous network includes systems produced by different manufacturers and/or running different operating systems.

hexadecimal number

A base 16 number. Hexadecimal (or hex) numbers are composed of the hexadecimal digits 09 and AF. See Table G-1 (next page).

HFS+

Hierarchical File System (new). HFS+ is the native OS X filesystem. Introduced in Mac OS 8.1, HFS+ is based on the older HFS.

HFSX

HFS+ Extended. The case sensitive version of HFS+, used primarily on Mac OS X Server.

hidden file

See invisible file on page 938.

hierarchy

An organization with a few things, or thingone at the topand with several things below each other thing. An inverted tree structure. Examples in computing include a file tree where each directory may contain files or other directories, a hierarchical network, and a class hierarchy in object-oriented programming.FOLDOC Refer to "The Hierarchical Filesystem" on page 72.

history

A shell mechanism that enables you to modify and reexecute recent commands.

home directory

The directory that is your working directory when you first log in. The pathname of this directory is stored in the HOME shell variable.

hover

To leave the mouse pointer stationary for a moment over an object. In many cases hovering displays a tooltip (page 958).

Table G-1. Decimal, octal, and hexadecimal numbers

Decimal

Octal

Hex

1

1

1

2

2

2

3

3

3

4

4

4

5

5

5

6

6

6

7

7

7

8

10

8

9

11

9

10

12

A

11

13

B

12

14

C

13

15

D

14

16

E

15

17

F

16

20

10

17

21

11

18

22

12

19

23

13

20

24

14

21

25

15

31

37

1F

32

40

20

33

41

21

64

100

40

96

140

60

100

144

64

128

200

80

254

376

FE

255

377

FF

256

400

100

257

401

101

HTML

Hypertext Markup Language. A hypertext document format used on the World Wide Web. Tags, which are embedded in the text, consist of a less than sign (<), a directive, zero or more parameters, and a greater than sign (>). Matched pairs of directives, such as <TITLE> and </TITLE>, delimit text that is to appear in a special place or style.FOLDOC For more information on HTML, go to www.htmlhelp.com/faq/html/all.html.

HTTP

Hypertext Transfer Protocol. The client/server TCP/IP protocol used on the World Wide Web for the exchange of HTML documents.

hub

A multiport repeater. A hub rebroadcasts all packets it receives on all ports. This term is frequently used to refer to small hubs and switches, regardless of the device's intelligence. It is a generic term for a layer 2 shared-media networking device. Today the term hub is sometimes used to refer to small intelligent devices, although that was not its original meaning. Contrast with network switch (page 945).

hypertext

A collection of documents/nodes containing (usually highlighted or underlined) cross-references or links, which, with the aid of an interactive browser program, allow the reader to move easily from one document to another.FOLDOC

Hypertext Markup Language

See HTML.

Hypertext Transfer Protocol

See HTTP.

i/o device

Input/output device. See device on page 929.

IANA

Internet Assigned Numbers Authority. A group that maintains a database of all permanent, registered system services (www.iana.org).

ICMP

Internet Control Message Protocol. A type of network packet that carries only messages, no data.

icon

In a GUI, a small picture representing a file, directory, action, program, and so on. When you click an icon, an action, such as opening a window and starting a program or displaying a directory or Web site, takes place. From miniature religious statues.FOLDOC

iconify

The process of changing a window into an icon. Contrast with restore (page 951).

ignored window

A state in which a window has no decoration and therefore no buttons or titlebar to control it with.

indentation

See indention.

indention

The blank space between the margin and the beginning of a line that is set in from the margin.

inode

A data structure (page 929) that contains information about a file. An inode for a file contains the file's length, the times the file was last accessed and modified, the time the inode was last modified, owner and group IDs, access privileges, number of links, and pointers to the data blocks that contain the file itself. Each directory entry associates a filename with an inode. Although a single file may have several filenames (one for each link), it has only one inode. Only some filesystems have inodes. HFS+ (page 935) filesystems, for example, do not have inodes while UFS (page 959) filesystems do.

input

Information that is fed to a program from a terminal or other file. See standard input on page 955.

installation

A computer at a specific location. Some aspects of the Mac OS X system are installation dependent. Also site.

interactive

A program that allows ongoing dialog with the user. When you give commands in response to shell prompts, you are using the shell interactively. Also, when you give commands to utilities, such as vim and mail, you are using the utilities interactively.

interface

The meeting point of two subsystems. When two programs work together, their interface includes every aspect of either program that the other deals with. The user interface (page 959) of a program includes every program aspect the user comes into contact with: the syntax and semantics involved in invoking the program, the input and output of the program, and its error and informational messages. The shell and each of the utilities and built-in commands have a user interface.

International Organization for Standardization

See ISO on page 939.

internet

A large network that encompasses other, smaller networks.

Internet

The largest internet in the world. The Internet (uppercase "I") is a multilevel hierarchy composed of backbone networks (ARPANET, NSFNET, MILNET, and others), midlevel networks, and stub networks. These include commercial (.com or .co), university (.ac or .edu), research (.org or .net), and military (.mil) networks and span many different physical networks around the world with various protocols, including the Internet Protocol (IP). Outside the United States, country code domains are popular (.us, .es, .mx, .de, and so forth), although you will see them used within the United States as well.

Internet Protocol

See IP.

Internet Service Provider

See ISP on page 939.

intranet

An inhouse network designed to serve a group of people such as a corporation or school. The general public on the Internet does not have access to the intranet.

invisible file

A file whose filename starts with a period. These files are called invisible because the ls utility does not normally list them. Use the a option of ls to list all files, including invisible ones. The shell does not expand a leading asterisk (*) in an ambiguous file reference to match the filename of an invisible file. Also hidden file.

Under OS X, also used to refer to a file with the invisible file attributes (page 932).

IP

Internet Protocol. The network layer for TCP/IP. IP is a best-effort, packet-switching, connectionless protocol (page 927) that provides packet routing, fragmentation, and reassembly through the data link layer. IPv4 is slowly giving way to IPv6.FOLDOC

IP address

Internet Protocol address. A four-part address associated with a particular network connection for a system using the Internet Protocol (IP). A system that is attached to multiple networks that use the IP will have a different IP address for each network interface.

IP multicast

See multicast on page 943.

IP spoofing

A technique used to gain unauthorized access to a computer. The would-be intruder sends messages to the target machine. These messages contain an IP address indicating that the messages are coming from a trusted host. The target machine responds to the messages, giving the intruder (privileged) access to the target.

IPC

Interprocess communication. A method to communicate specific information between programs.

IPv4

IP version 4. See IP and IPv6.

IPv6

IP version 6. The next generation of Internet Protocol, which provides a much larger address space (2128 bits versus 232 bits for IPv4) that is designed to accommodate the rapidly growing number of Internet addressable devices. IPv6 also has built-in autoconfiguration, enhanced security, better multicast support, and many other features.

ISDN

Integrated Services Digital Network. A set of communications standards that allows a single pair of digital or standard telephone wires to carry voice, data, and video at a rate of 64 kilobits per second.

ISO

International Organization for Standardization. A voluntary, nontreaty organization founded in 1946. It is responsible for creating international standards in many areas, including computers and communications. Its members are the national standards organizations of 89 countries, including the American National Standards Institute.FOLDOC

ISO9660

The ISO standard defining a filesystem for CD-ROMs.

ISP

Internet service provider. Provides Internet access to its customers.

job control

A facility that enables you to move commands from the foreground to the background and vice versa. Job control enables you to stop commands temporarily.

journaling filesystem

A filesystem that maintains a noncached log file, or journal, which records all transactions involving the filesystem. When a transaction is complete, it is marked as complete in the log file.

The log file results in greatly reduced time spent recovering a filesystem after a crash, making it particularly valuable in systems where high availability is an issue.

JPEG

Joint Photographic Experts Group. This committee designed the standard algorithm for compressing images. JPEG is intended for compressing either full-color or gray-scale digital images of natural, real-world scenes and does not work as well on nonrealistic images, such as cartoons or line drawings. Filename extensions: .jpg, .jpeg.FOLDOC

justify

To expand a line of type in the process of formatting text. A justified line has even margins. A line is justified by increasing the space between words and sometimes between letters on the line.

Kerberos

An MIT-developed security system that authenticates users and machines. It does not provide authorization to services or databases; it establishes identity at logon, which is used throughout the session. Once you are authenticated, you can open as many terminals, windows, services, or other network accesses as you like until your session expires.

kernel

The part of the operating system that allocates machine resources, including memory, disk space, and CPU (page 928) cycles, to all other programs that run on a computer. The kernel includes the low-level hardware interfaces (drivers) and manages processes (page 948), the means by which Mac OS X executes programs.

kernelspace

The part of memory (RAM) where the kernel resides. Code running in kernelspace has full access to hardware and all other processes in memory. See the KernelAnalysis-HOWTO.

key binding

A keyboard key is said to be bound to the action that results from pressing it. Typically keys are bound to the letters that appear on the keycaps: When you press A, an A appears on the screen. Key binding usually refers to what happens when you press a combination of keys, one of which is CONTROL, ALT, META, OPTION, or SHIFT, or when you press a series of keys, the first of which is typically ESCAPE.

keyboard

A hardware input device consisting of a number of mechanical buttons (keys) that the user presses to input characters to a computer. By default a keyboard is connected to standard input of a shell.FOLDOC

kilo-

In the binary system, the prefix kilo- multiplies by 210 (i.e., 1,024). Kilobit and kilobyte are common uses of this prefix. Abbreviated as k.

Korn Shell

ksh. A command processor, developed by David Korn at AT&T Bell Laboratories, that is compatible with the Bourne Shell but includes many extensions. See also shell on (page 953)

LAN

Local area network. A network that connects computers within a localized area (such as a single site, building, or department).

large number

Go to mathworld.wolfram.com/LargeNumber.html for a comprehensive list.

LDAP

Lightweight Directory Access Protocol. A simple protocol for accessing online directory services. Traditionally LDAP has been used to access information such as email directories; in some cases, it can be used as an alternative for services such as NIS. Given a name, many mail clients can use LDAP to discover the corresponding email address. See directory service on page 930.

leaf

In a tree structure, the end of a branch that cannot support other branches. When the Mac OS X filesystem hierarchy is conceptualized as a tree, files that are not directories are leaves. See node on page 945.

least privilege, concept of

Mistakes that Superuser makes can be much more devastating than those made by an ordinary user. When you are working on the computer, especially when you are working as the system administrator, always perform any task using the least privilege possible. If you can perform a task logged in as an ordinary user, do so. If you must be logged in as Superuser, do as much as you can as an ordinary user, log in as or give an su or sudo command so that you are working as Superuser, do as much of the task that has to be done as Superuser, and revert to being an ordinary user as soon as you can.

Because you are more likely to make a mistake when you are rushing, this concept becomes more important when you have less time to apply it.

Lightweight Directory Access Protocol

See LDAP.

link

A pointer to a file. Two kinds of links exist: hard links and symbolic (soft) links. A hard link associates a filename with a place on the disk where the contents of the file is located. A symbolic link associates a filename with the pathname of a hard link to a file. See hard link (page 934) and symbolic link (page 957). Sometimes called an alias by Macintosh users and documentation.

Linux-PAM

See PAM on page 946.

Linux-Pluggable Authentication Modules

See PAM on page 946.

loadable kernel module

See loadable module.

loadable module

A portion of the operating system that controls a special device and that can be loaded automatically into a running kernel as needed to access that device.

local area network

See LAN on page 940.

locale

The language; date, time, and currency formats; character sets; and so forth that pertain to a geopolitical place or area. For example, en_US specifies English as spoken in the United States and dollars; en_UK specifies English as spoken in the United Kingdom and pounds. See the locale man page for more information.

log in

To gain access to a computer system by responding correctly to the login: and Password: prompts. Also log on, login.

log out

To end your session by exiting from your login shell. Also log off.

logical expression

A collection of strings separated by logical operators (>,>=,=,!=,<=, and <) that can be evaluated as true or false. Also Boolean expression.

.login file

A file in a user's home directory that the TC Shell executes when you log in. You can use this file to set environment variables and to run commands that you want executed at the beginning of each session.

login name

See username on page 960.

login shell

The shell that you are using when you log in. The login shell can fork other processes that can run other shells, utilities, and programs.

.logout file

A file in a user's home directory that the TC Shell executes when you log out, assuming that the TC Shell is your login shell. You can put in the .logout file commands that you want run each time you log out.

MAC address

Media Access Control address. The unique hardware address of a device connected to a shared network medium. Each Ethernet adapter has a globally unique MAC address in ROM. MAC addresses are 6 bytes long, enabling 2566 (about 300 trillion) possible addresses or 65,536 addresses for each possible IPv4 address.

A MAC address performs the same role for Ethernet that an IP address performs for TCP/IP: It provides a unique way to identify a host.

machine collating sequence

The sequence in which the computer orders characters. The machine collating sequence affects the outcome of sorts and other procedures that put lists in alphabetical order. Many computers use ASCII codes so their machine collating sequences correspond to the ordering of the ASCII codes for characters.

macro

A single instruction that a program replaces by several (usually more complex) instructions. The C compiler recognizes macros, which are defined using a #define instruction to the preprocessor.

magic number

A magic number, which occurs in the first 512 bytes of a binary file, is a 1-, 2-, or 4-byte numeric value or character string that uniquely identifies the type of file (much like a DOS 3-character filename extension). See /usr/share/file/magic and the magic man page (5) for more information.

main memory

Random access memory (RAM), an integral part of the computer. Although disk storage is sometimes referred to as memory, it is never referred to as main memory.

major device number

A number assigned to a class of devices, such as terminals, printers, or disk drives. Using the ls utility with the l option to list the contents of the /dev directory displays the major and minor device numbers of many devices (as major, minor).

MAN

Metropolitan area network. A network that connects computers and LANs (page 940) at multiple sites in a small regional area, such as a city.

masquerade

To appear to come from one domain or IP address when actually coming from another. Said of a packet or message (sendmail). See also NAT on page 944.

MD5

Message Digest 5. A one-way hash function (page 945).

MDA

Mail delivery agent. One of the three components of a mail system; the other two are the MTA and MUA. An MDA accepts inbound mail from an MTA and delivers it to a local user.

mega-

In the binary system, the prefix mega- multiplies by 220 (i.e., 1,048,576). Megabit and megabyte are common uses of this prefix. Abbreviated as M.

menu

A list from which the user may select an operation to be performed. This selection is often made with a mouse or other pointing device under a GUI but may also be controlled from the keyboard. Very convenient for beginners, menus show which commands are available and facilitate experimenting with a new program, often reducing the need for user documentation. Experienced users usually prefer keyboard commands, especially for frequently used operations, because they are faster to use.FOLDOC

merge

To combine two ordered lists so that the resulting list is still in order. The sort utility can merge files.

META key

On the keyboard, a key that is labeled META or ALT. Use this key as you would the SHIFT key. While holding it down, press another key. The emacs editor makes extensive use of the META key. On the Mac, often labeled OPTION. For more information see the "Activating the META key" tip on page 31.

metacharacter

A character that has a special meaning to the shell or another program in a particular context. Metacharacters are used in the ambiguous file references recognized by the shell and in the regular expressions recognized by several utilities. You must quote a metacharacter if you want to use it without invoking its special meaning. See regular character (page 950) and special character (page 954).

metadata

Data about data. In data processing, metadata is definitional data that provides information about, or documentation of, other data managed within an application or environment.

For example, metadata can document data about data elements or attributes (name, size, data type, and so on), records or data structures (page 929) (length, fields, columns, and so on), and data itself (where it is located, how it is associated, who owns it, and so on). Metadata can include descriptive information about the context, quality and condition, or characteristics of the data.FOLDOC

Examples of metadata on Mac OS X systems include file attributes (page 932), ACLs (page 920), and traditional UNIX access permissions (page 87).

metropolitan area network

See MAN.

MIME

Multipurpose Internet Mail Extension. Originally used to describe how specific types of files that were attached to email were to be handled. Today MIME types describe how a file is to be opened or worked with, based on its filename extension.

minimize

See iconify on page 937.

minor device number

A number assigned to a specific device within a class of devices. See major device number on page 942.

modem

Modulator/demodulator. A peripheral device that modulates digital data into analog data for transmission over a voice-grade telephone line. Another modem demodulates the data at the other end.

module

See loadable module on page 940.

mount

To make a filesystem accessible to system users. When a filesystem is not mounted, you cannot read from or write to files it contains.

mount point

A directory that you mount a local or remote filesystem on.

mouse

A device you use to point to a particular location on a display screen, typically so you can choose a menu item, draw a line, or highlight some text. You control a pointer on the screen by sliding a mouse around on a flat surface; the position of the pointer moves relative to the movement of the mouse. You select items by pressing one or more buttons on the mouse.

mouse pointer

In a GUI, a marker that moves in correspondence with the mouse. It is usually a small arrow. Differs from the cursor (page 928).

mouseover

The action of passing the mouse pointer over an icon or other object on the screen.

MTA

Mail transfer agent. One of the three components of a mail system; the other two are the MDA and MUA. An MTA accepts mail from users and MTAs.

MUA

Mail user agent. One of the three components of a mail system; the other two are the MDA and MTA. An MUA is an end-user mail program such as Kmail, mutt, or Outlook.

multiboot specification

Specifies an interface between a boot loader and an operating system. With compliant boot loaders and operating systems, any boot loader should be able to load any operating system. The object of this specification is to ensure that different operating systems will work on a single machine. For more information, go to odin-os.source-forge.net/guides/multiboot.html.

multicast

A multicast packet has one source and multiple destinations. In multicast, source hosts register at a special address to transmit data. Destination hosts register at the same address to receive data. In contrast to broadcast (page 924), which is LAN-based, multicast traffic is designed to work across routed networks on a subscription basis. Multicast reduces network traffic by transmitting a packet one time, with the router at the end of the path breaking it apart as needed for multiple recipients.

multitasking

A computer system that allows a user to run more than one job at a time. A multitasking system, such as Mac OS X, allows you to run a job in the background while running a job in the foreground.

multiuser system

A computer system that can be used by more than one person at a time. Mac OS X is a multiuser operating system. Contrast with single-user system (page 953).

NAT

Network Address Translation. A scheme that enables a LAN to use one set of IP addresses internally and a different set externally. The internal set is for LAN (private) use. The external set is typically used on the Internet and is Internet unique. NAT provides some privacy by hiding internal IP addresses and allows multiple internal addresses to connect to the Internet through a single external IP address. See also masquerade on page 942.

NBT

NetBIOS over TCP/IP. A protocol that supports NetBIOS services in a TCP/IP environment. Also NetBT.

NetBIOS

Network Basic Input/Output System. An API (page 921) for writing network-aware applications.

netboot

To boot a computer over the network (as opposed to booting from a local disk).

netiquette

The conventions of etiquettethat is, polite behaviorrecognized on Usenet and in mailing lists, such as not (cross-)posting to inappropriate groups and refraining from commercial advertising outside the business groups.

The most important rule of netiquette is "Think before you post." If what you intend to post will not make a positive contribution to the newsgroup and be of interest to several readers, do not post it. Personal messages to one or two individuals should not be posted to newsgroups; use private email instead.FOLDOC

netmask

A 32-bit mask (for IPv4), that shows how an Internet address is to be divided into network, subnet, and host parts. The netmask has ones in the bit positions in the 32-bit address that are to be used for the network and subnet parts and zeros for the host part. The mask should contain at least the standard network portion (as determined by the address class). The subnet field should be contiguous with the network portion.FOLDOC

network address

The network portion (netid) of an IP address. For a class A network, it is the first byte, or segment, of the IP address; for a class B network, it is the first two bytes; and for a class C network, it is the first three bytes. In each case the balance of the IP address is the host address (hostid). Assigned network addresses are globally unique within the Internet. Also network number.

Network Filesystem

See NFS on page 945.

Network Information Service

See NIS.

network number

See network address.

network segment

A part of an Ethernet or other network on which all message traffic is common to all nodes; that is, it is broadcast from one node on the segment and received by all others. This commonality normally occurs because the segment is a single continuous conductor. Communication between nodes on different segments is via one or more routers.FOLDOC

network switch

A connecting device in networks. Switches are increasingly replacing shared media hubs in an effort to increase bandwidth. For example, a 16-port 10BaseT hub shares the total 10 megabits per second bandwidth with all 16 attached nodes. By replacing the hub with a switch, both sender and receiver can take advantage of the full 10 megabits per second capacity. Each port on the switch can give full bandwidth to a single server or client station or to a hub with several stations. Network switch refers to a device with intelligence. Contrast with hub (page 936).

Network Time Protocol

See NTP.

NFS

Network Filesystem. A remote filesystem designed by Sun Microsystems, available on computers from most UNIX system vendors.

NIC

Network interface card (or controller). An adapter circuit board installed in a computer to provide a physical connection to a network.FOLDOC

NIS

Network Information Service. A distributed service built on a shared database to manage system-independent information (such as login names and passwords).

NIS domain name

A name that describes a group of systems that share a set of NIS files. Different from domain name (page 930).

NNTP

Network News Transfer Protocol.

node

In a tree structure, the end of a branch that can support other branches. When the Mac OS X filesystem hierarchy is conceptualized as a tree, directories are nodes. See leaf on page 940.

nonprinting character

See control character on page 927. Also nonprintable character.

nonvolatile storage

A storage device whose contents are preserved when its power is off. Also NVS and persistent storage. Some examples are CD-ROM, paper punch tape, hard disk, ROM (page 951), PROM (page 948), EPROM (page 931), and EEPROM (page 931). Contrast with RAM (page 950).

NTP

Network Time Protocol. Built on top of TCP/IP, NTP maintains accurate local time by referring to known accurate clocks on the Internet.

null string

A string that could contain characters but does not. A string of zero length.

octal number

A base 8 number. Octal numbers are composed of the digits 07, inclusive. Refer to Table G-1 on page 936.

one-way hash function

A one-way function that takes a variable-length message and produces a fixed-length hash. Given the hash, it is computationally infeasible to find a message with that hash; in fact, you cannot determine any usable information about a message with that hash. Also message digest function. See also hash (page 935).

Open Firmware

A standardized boot firmware (page 933) used on Macintosh and Sun computers. Open Firmware handles device access and is the primary boot loader (page 923) for Mac OS X. Open Firmware is believed to be the only IEEE standard ever to have its own song.

OpenSSH

A free version of the SSH (secure shell) protocol suite that replaces TELNET, rlogin, and more with secure programs that encrypt all communicationeven passwordsover a network.

operating system

A control program for a computer that allocates computer resources, schedules tasks, and provides the user with a way to access resources.

option

A command line argument that modifies the effects of a command. Options are usually preceded by hyphens on the command line and traditionally have single-character names (such as h or n). Some commands allow you to group options following a single hyphen (for example, -hn). GNU utilities frequently have two arguments that do the same thing: a single-character argument and a longer, more descriptive argument that is preceded by two hyphens (such as --show-all and --invert-match).

ordinary file

A file that is used to store a program, text, or other user data. See directory (page 930) and device file (page 930).

output

Information that a program sends to the terminal or another file. See standard output on page 955.

P2P

Peer-to-Peer. A network that does not divide nodes into clients and servers. Each computer on a P2P network can fulfill the roles of client and server. In the context of a file-sharing network, this ability means that once a node has downloaded (part of) a file, it can act as a server. BitTorrent implements a P2P network.

packet

A unit of data sent across a network. Packet is a generic term used to describe a unit of data at any layer of the OSI protocol stack, but it is most correctly used to describe network or application layer data units ("application protocol data unit," APDU).FOLDOC See also frame (page 933) and datagram (page 929).

packet filtering

A technique used to block network traffic based on specified criteria, such as the origin, destination, or type of each packet. See also firewall (page 933).

packet sniffer

A program or device that monitors packets on a network. See sniff on page 954.

pager

A utility that allows you to view a file one screen at a time (for example, less and more).

paging

The process by which virtual memory is maintained by the operating system. The contents of process memory is moved (paged out) to the swap space (page 956) as needed to make room for other processes.

PAM

Linux-PAM or Linux-Pluggable Authentication Modules. These modules allow a system administrator to determine how various applications authenticate users. PAM did not originate on Linux, but Mac OS X uses the version developed for Linux.

parent process

A process that forks other processes. See process (page 948) and child process (page 926).

partition

A section of a (hard) disk that has a name so you can address it separately from other sections. A disk partition can hold a filesystem or another structure, such as the swap area. Under DOS and Windows, partitions (and sometimes whole disks) are labeled C:, D:, and so on. Also disk partition and slice.

passive FTP

Allows FTP to work through a firewall by allowing the flow of data to be initiated and controlled by the client FTP program instead of the server. Also called PASV FTP because it uses the FTP PASV command.

passphrase

A string of words and characters that you type in to authenticate yourself. A passphrase differs from a password only in length. A password is usually short6 to 10 characters. A passphrase is usually much longerup to 100 characters or more. The greater length makes a passphrase harder to guess or reproduce than a password and therefore more secure.FOLDOC

password

To prevent unauthorized access to a user's account, an arbitrary string of characters chosen by the user or system administrator and used to authenticate the user when attempting to log in.FOLDOC See also passphrase.

PASV FTP

See passive FTP on page 946.

pathname

A list of directories separated by slashes (/) and ending with the name of a file, which can be a directory. A pathname is used to trace a path through the file structure to locate or identify a file.

pathname, last element of a

The part of a pathname following the final /, or the whole filename if there is no /. A simple filename. Also basename.

pathname element

One of the filenames that forms a pathname.

peripheral device

See device on page 929.

persistent

Data that is stored on nonvolatile media, such as a hard disk.

phish

An attempt to trick users into revealing or sharing private information, especially passwords or financial information. The most common form is email purporting to be from a bank or vendor that requests that a user fill out a form to "update" an account. Generally sent as spam (page 954).

physical device

A tangible device, such as a disk drive, that is physically separate from other, similar devices.

PID

Process identification, usually followed by the word number. Mac OS X assigns a unique PID number as each process is initiated.

pipe

A connection between programs such that standard output of one program is connected to standard input of the next. Also pipeline.

pixel

The smallest element of a picture, typically a single dot on a display screen.

plaintext

Text that is not encrypted. Also cleartext. Contrast with ciphertext (page 926).

Pluggable Authentication Modules

See PAM on page 946.

point-to-point link

A connection limited to two endpoints, such as the connection between a pair of modems.

port

A logical channel or channel endpoint in a communications system. The TCP (page 957) and UDP (page 959) transport layer protocols used on Ethernet use port numbers to distinguish between different logical channels on the same network interface on the same computer.

The /etc/services file (see the beginning of this file for more information) or the services database specifies a unique port number for each application program. The number links incoming data to the correct service (program). Standard, well-known ports are used by everyone: Port 80 is used for HTTP (Web) traffic. Some protocols, such as TELNET and HTTP (which is a special form of TELNET), have default ports specified as mentioned earlier but can use other ports as well.FOLDOC

port forwarding

The process by which a network port on one computer is transparently connected to a port on another computer. If port X is forwarded from system A to system B, any data sent to port X on system A is sent to system B automatically. The connection can be between different ports on the two systems.

portmapper

A server that converts TCP/IP port numbers into RPC (page 952) program numbers.

printable character

One of the graphic characters: a letter, number, or punctuation mark. Contrast with a nonprintable, or CONTROL, character. Also printing character.

private address space

IANA (page 937) has reserved three blocks of IP addresses for private internets or LANs:

10.0.0.0 - 10.255.255.255

172.16.0.0 - 172.31.255.255

192.168.0.0 - 192.168.255.255

You can use these addresses without coordinating with anyone outside of your LAN (you do not have to register the system name or address). Systems using these IP addresses cannot communicate directly with hosts using the global address space but must go through a gateway. Because private addresses have no global meaning, routing information is not stored by DNSs and most ISPs reject privately addressed packets. Make sure that your router is set up not to forward these packets onto the Internet.

privileged port

A port (page 947) with a number less than 1,024. On Mac OS X and other UNIX-like systems, only root can bind to a privileged port. Any user on Windows 98 and earlier Windows systems can bind to any port. Also reserved port.

procedure

A sequence of instructions for performing a particular task. Most programming languages, including machine languages, enable a programmer to define procedures that allow the procedure code to be called from multiple places. Also subroutine.FOLDOC

process

The execution of a command by Mac OS X. See "Processes" on page 293.

.profile file

A startup file in a user's home directory that the Bourne Again Shell executes when you log in. The TC Shell executes .login instead. You can use the .profile file to run commands, set variables, and define functions.

program

A sequence of executable computer instructions contained in a file. Mac OS X utilities, applications, and shell scripts are all programs. Whenever you run a command that is not built into a shell, you are executing a program.

PROM

Programmable readonly memory. A kind of nonvolatile storage. ROM (page 951) that can be written to using a PROM programmer.

prompt

A cue from a program, usually displayed on the screen, indicating that it is waiting for input. The shell displays a prompt, as do some of the interactive utilities, such as mail. By default the Bourne Again and Z Shells use a dollar sign ($) as a prompt, and the TC Shell uses a percent sign (%).

protocol

A set of formal rules describing how to transmit data, especially across a network. Low-level protocols define the electrical and physical standards, bit and byte ordering, and transmission, error detection, and correction of the bit stream. High-level protocols deal with data formatting, including message syntax, terminal-to-computer dialog, character sets, and sequencing of messages.FOLDOC

proxy

A service that is authorized to act for a system while not being part of that system. See also proxy gateway and proxy server.

proxy gateway

A computer that separates clients (such as browsers) from the Internet, working as a trusted agent that accesses the Internet on their behalf. A proxy gateway passes a request for data from an Internet service, such as HTTP from a browser/client, to a remote server. The data that the server returns goes back through the proxy gateway to the requesting service. A proxy gateway should be transparent to the user.

A proxy gateway often runs on a firewall (page 933) system and acts as a barrier to malicious users. It hides the IP addresses of the local computers inside the firewall from Internet users outside the firewall.

You can configure browsers, such as Mozilla and Netscape, to use a different proxy gateway or to use no proxy for each URL access method including FTP, netnews, SNMP, HTTPS, and HTTP. See also proxy.

proxy server

A proxy gateway that usually includes a cache (page 925) that holds frequently used Web pages so that the next request for that page is available locally (and therefore more quickly). The terms proxy server and proxy gateway are frequently interchanged so that the use of cache does not rest exclusively with the proxy server. See also proxy.

Python

A simple, high-level, interpreted, object-oriented, interactive language that bridges the gap between C and shell programming. Suitable for rapid prototyping or as an extension language for C applications, Python supports packages, modules, classes, user-defined exceptions, a good C interface, and dynamic loading of C modules. It has no arbitrary restrictions. For more information, see www.python.org FOLDOC

quote

When you quote a character, you take away any special meaning that it has in the current context. You can quote a character by preceding it with a backslash. When you are interacting with the shell, you can also quote a character by surrounding it with single quotation marks. For example, the command echo \* or echo '*' displays *. The command echo * displays a list of the files in the working directory. See ambiguous file reference (page 920), metacharacter (page 942), regular character (page 950), regular expression (page 950), and special character (page 954). See also escape on page 931.

radio button

One of a group of buttons similar to those used to select the station on a radio. Only one button can be selected at a time.

RAID

Redundant array of inexpensive/independent disks. Two or more (hard) disk drives used in combination to improve fault tolerance and performance. RAID can be implemented in hardware or software.

RAM

Random access memory. A kind of volatile storage. A data storage device for which the order of access to different locations does not affect the speed of access. Contrast with a hard disk or tape drive, which provides quicker access to sequential data because accessing a nonsequential location requires physical movement of the storage medium and/or read/write head rather than just electronic switching. Contrast with nonvolatile storage (page 945).FOLDOC

RAM disk

RAM that is made to look like a floppy diskette or hard disk. A RAM disk is frequently used as part of the boot (page 923) process.

RAS

Remote access server. In a network, a computer that provides access to remote users via analog modem or ISDN connections. RAS includes the dial-up protocols and access control (authentication). It may be a regular fileserver with remote access software or a proprietary system, such as Shiva's LANRover. The modems may be internal or external to the device.

RDF

Resource Description Framework. Being developed by W3C (the main standards body for the World Wide Web), a standard that specifies a mechanism for encoding and transferring metadata (page 942). RDF does not specify what the metadata should or can be. It can integrate many kinds of applications and data, using XML as an interchange syntax. Examples of the data that can be integrated include library catalogs and worldwide directories; syndication and aggregation of news, software, and content; and collections of music and photographs. Go to www.w3.org/RDF for more information.

redirection

The process of directing standard input for a program to come from a file rather than from the keyboard. Also, directing standard output or standard error to go to a file rather than to the screen.

reentrant

Code that can have multiple simultaneous, interleaved, or nested invocations that do not interfere with one another. Noninterference is important for parallel processing, recursive programming, and interrupt handling.

It is usually easy to arrange for multiple invocations (that is, calls to a subroutine) to share one copy of the code and any readonly data. For the code to be reentrant, however, each invocation must use its own copy of any modifiable data (or synchronized access to shared data). This goal is most often achieved by using a stack and allocating local variables in a new stack frame for each invocation. Alternatively, the caller may pass in a pointer to a block of memory that that invocation can use (usually for output), or the code may allocate some memory on a heap, especially if the data must survive after the routine returns.

Reentrant code is often found in system software, such as operating systems and teleprocessing monitors. It is also a crucial component of multithreaded programs, where the term thread-safe is often used instead of reentrant.FOLDOC

regular character

A character that always represents itself in an ambiguous file reference or another type of regular expression. Contrast with special character.

regular expression

A stringcomposed of letters, numbers, and special symbolsthat defines one or more strings. See Appendix A.

relative pathname

A pathname that starts from the working directory. Contrast with absolute pathname (page 920).

remote access server

See RAS on page 950.

remote filesystem

A filesystem on a remote computer that has been set up so that you can access (usually over a network) its files as though they were stored on your local computer's disks. An example of a remote filesystem is NFS.

remote procedure call

See RPC on page 952

resolver

The TCP/IP library software that formats requests to be sent to the DNS (page 930) for hostname-to-Internet address conversion.FOLDOC

Resource Description Framework

See RDF on page 950.

resource fork

A fork (page 933) of a file on an HFS+ filesystem (page 935). The resource fork is a database that allows random access to resources. Contrast with data fork (page 929).

restore

The process of turning an icon into a window. Contrast with iconify (page 937)

return code

See exit status on page 932.

RFC

Request for comments. Begun in 1969, one of a series of numbered Internet informational documents and standards widely followed by commercial software and freeware in the Internet and UNIX/Linux communities. Few RFCs are standards but all Internet standards are recorded in RFCs. Perhaps the single most influential RFC has been RFC 822, the Internet electronic mail format standard.

The RFCs are unusual in that they are floated by technical experts acting on their own initiative and reviewed by the Internet at large rather than being formally promulgated through an institution such as ANSI. For this reason they remain known as RFCs, even after they are adopted as standards. The RFC tradition of pragmatic, experience-driven, after-the-fact standard writing done by individuals or small working groups has important advantages over the more formal, committee-driven process typical of ANSI or ISO. For a complete list of RFCs, go to www.rfc-editor.org.FOLDOC

roam

To move a computer between wireless access points (page 961) on a wireless network without the user or applications being aware of the transition. Moving between access points typically results in some packet loss, although this loss is transparent to programs that use TCP.

ROM

Readonly memory. A kind of nonvolatile storage. A data storage device that is manufactured with fixed contents. In general, ROM describes any storage system whose contents cannot be altered, such as a phonograph record or printed book. When used in reference to electronics and computers, ROM describes semiconductor integrated circuit memories, of which several types exist, and CD-ROM.

ROM is nonvolatile storageit retains its contents even after power has been removed. ROM is often used to hold programs for embedded systems, as these usually have a fixed purpose. ROM is also used for storage of the BIOS (page 923) in a computer. Contrast with RAM (page 950).FOLDOC

root directory

The ancestor of all directories and the start of all absolute pathnames. The name of the root directory is /.

root filesystem

The filesystem that is available when the system is brought up in single-user mode. The name of this filesystem is always /. You cannot unmount or mount the root filesystem. You can remount root to change its mount options. Also the startup disk (page 955) or startup volume.

root login

Usually the login name of Superuser (page 956).

root (user)

Another name for Superuser (page 956).

rotate

When a file, such as a log file, gets indefinitely larger, you must keep it from taking up too much space on the disk. Because you may need to refer to the information in the log files in the near future, it is generally not a good idea to delete the contents of the file until it has aged. Instead you can periodically save the current log file under a new name and create a new, empty file as the current log file. You can keep a series of these files, renaming each as a new one is saved. You will then rotate the files. For example, you might remove xyzlog.4, xyzlog.3 [image] xyzlog.4, xyzlog.2 [image] xyzlog.3, xyzlog.1 [image] xyzlog.2, xyzlog [image] xyzlog.1, and create a new xyzlog file. By the time you remove xyzlog.4, it will not contain any information more recent than you want to remove.

router

A device (often a computer) that is connected to more than one similar type of network to pass data between them. See gateway on page 934.

RPC

Remote procedure call. A call to a procedure (page 948) that acts transparently across a network. The procedure itself is responsible for accessing and using the network. The RPC libraries make sure that network access is transparent to the application. RPC runs on top of TCP/IP or UDP/IP.

RSA

A public key encryption technology that is based on the lack of an efficient way to factor very large numbers. Because of this lack, it takes an extraordinary amount of computer processing time and power to deduce an RSA key. The RSA algorithm is the de facto standard for data sent over the Internet.

run

To execute a program.

Samba

A free suite of programs that implement the Server Message Block (SMB) protocol. See SMB (page 953).

schema

Within a GUI, a pattern that helps you see and interpret the information that is presented in a window, making it easier to understand new information that is presented using the same schema.

scroll

To move lines on a terminal or window up and down or left and right.

scrollbar

A widget found in graphical user interfaces that controls (scrolls) which part of a document is visible in the window. A window can have a horizontal scroll bar, a vertical scroll bar (more common), or both.FOLDOC

server

A powerful centralized computer (or program) designed to provide information to clients (smaller computers or programs) on request.

session

The lifetime of a process. For a desktop, it is the desktop session manager. For a character-based terminal, it is the user's login shell process. A session may also be the sequence of events between when you start using a program, such as an editor, and when you finish.

setgid

When you execute a file that has setgid (set group ID) permission, the process executing the file takes on the privileges of the group the file belongs to. The ls utility shows setgid permission as an s in the group's executable position. See also setuid.

setuid

When you execute a file that has setuid (set user ID) permission, the process executing the file takes on the privileges of the owner of the file. As an example, if you run a setuid program that removes all the files in a directory, you can remove files in any of the file owner's directories, even if you do not normally have permission to do so. When the program is owned by root, you can remove files in any directory that root can remove files from. The ls utility shows setuid permission as an s in the owner's executable position. See also setgid.

sexillion

In the British system, 1036. In the American system, this number is named undecillion. See also large number (page 940).

share

A directory and the filesystem hierarchy below it that are shared with another system using SMB. Also Windows share (page 961).

shared network topology

A network, such as Ethernet, in which each packet may be seen by systems other than its destination system. Shared means that the network bandwidth is shared by all users.

shell

A Mac OS X system command processor. The three major shells are the Bourne Again Shell (page 924), the TC Shell (page 957), and the Z Shell (page 962).

shell function

A series of commands that the shell stores for execution at a later time. Shell functions are like shell scripts but run more quickly because they are stored in the computer's main memory rather than in files. Also, a shell function is run in the environment of the shell that calls it (unlike a shell script, which is typically run in a subshell).

shell script

An ASCII file containing shell commands. Also shell program.

signal

A very brief message that the UNIX system can send to a process, apart from the process's standard input. Refer to "trap: Catches a Signal" on page 577.

simple filename

A single filename containing no slashes (/). A simple filename is the simplest form of pathname. Also the last element of a pathname. Also basename (page 922).

single-user system

A computer system that only one person can use at a time. Contrast with multiuser system (page 944).

SMB

Server Message Block. Developed in the early 1980s by Intel, Microsoft, and IBM, SMB is a client/server protocol that is the native method of file and printer sharing for Windows. In addition, SMB can share serial ports and communications abstractions, such as named pipes and mail slots. SMB is similar to a remote procedure call (RPC, page 952) that has been customized for filesystem access. Also Microsoft Networking.FOLDOC

smiley

A character-based glyph (page 934), typically used in email, that conveys an emotion. The characters :-) in a message portray a smiley face (look at it sideways). Because it can be difficult to tell when the writer of an electronic message is saying something in jest or in seriousness, email users often use :-) to indicate humor. The two original smileys, designed by Scott Fahlman, were :-) and :-(. Also emoticon, smileys, and smilies. For more information search on smiley on the Internet.

smilies

See smiley.

SMTP

Simple Mail Transfer Protocol. A protocol used to transfer electronic mail between computers. It is a server-to-server protocol, so other protocols are used to access the messages. The SMTP dialog usually happens in the background under the control of a message transport system such as sendmail.FOLDOC

snap (windows)

As you drag a window toward another window or edge of the workspace, it can move suddenly so that it is adjacent to the other window/edge. Thus the window snaps into position.

sneakernet

Using hand-carried magnetic media to transfer files between machines.

sniff

To monitor packets on a network. A system administrator can legitimately sniff packets and a malicious user can sniff packets to obtain information such as usernames and passwords. See also packet sniffer (page 946).

SOCKS

A networking proxy protocol embodied in a SOCKS server, which performs the same functions as a proxy gateway (page 949) or proxy server (page 949). SOCKS works at the application level, requiring that an application be modified to work with the SOCKS protocol, whereas a proxy (page 949) makes no demands on the application.

SOCKSv4 does not support authentication or UDP proxy. SOCKSv5 supports a variety of authentication methods and UDP proxy.

sort

To put in a specified order, usually alphabetic or numeric.

SPACE character

A character that appears as the absence of a visible character. Even though you cannot see it, a SPACE is a printable character. It is represented by the ASCII code 32 (decimal). A SPACE character is considered a blank or whitespace (page 961).

spam

Posting irrelevant or inappropriate messages to one or more Usenet newsgroups or mailing lists in deliberate or accidental violation of netiquette (page 944). Also, sending large amounts of unsolicited email indiscriminately. This email usually promotes a product or service. Another common purpose of spam is to phish (page 947). Spam is the electronic equivalent of junk mail. From the Monty Python "Spam" song.FOLDOC

sparse file

A file that is large but takes up little disk space. The data in a sparse file is not dense (thus its name). Examples of sparse files are core files and dbm files.

spawn

See fork on page 933.

special character

A character that has a special meaning when it occurs in an ambiguous file reference or another type of regular expression, unless it is quoted. The special characters most commonly used with the shell are * and ?. Also metacharacter (page 942) and wildcard.

special file

See device file on page 930.

spinner

In a GUI, a type of text box (page 957) that holds a number you can change by typing over it or using the up and down arrows at the end of the box.

spoofing

See IP spoofing on page 938.

spool

To place items in a queue, each waiting its turn for some action. Often used when speaking about printers. Also used to describe the queue.

SQL

Structured Query Language. A language that provides a user interface to relational database management systems (RDBMS). SQL, the de facto standard, is also an ISO and ANSI standard and is often embedded in other programming languages.FOLDOC

square bracket

A left square bracket ([) or a right square bracket (]). These special characters define character classes in ambiguous file references and other regular expressions.

SSH Communications Security

The company that created the original SSH (secure shell) protocol suite (www.ssh.com). Mac OS X uses OpenSSH. See OpenSSH on page 946.

standard error

A file to which a program can send output. Usually only error messages are sent to this file. Unless you instruct the shell otherwise, it directs this output to the screen (that is, to the device file that represents the screen).

standard input

A file from which a program can receive input. Unless you instruct the shell otherwise, it directs this input so that it comes from the keyboard (that is, from the device file that represents the keyboard).

standard output

A file to which a program can send output. Unless you instruct the shell otherwise, it directs this output to the screen (that is, to the device file that represents the screen).

startup disk

The disk or volume used to boot (page 923) a system. Also root filesystem (page 952).

startup file

A file that the login shell runs when you log in. The Bourne Again and Z Shells run .pro-file, and the TC Shell runs .login. The TC Shell also runs .cshrc whenever a new TC Shell or a subshell is invoked. The Z Shell runs an analogous file whose name is identified by the ENV variable.

status line

The bottom (usually the twenty-fourth) line of the terminal. The vim editor uses the status line to display information about what is happening during an editing session.

sticky bit

An access permission bit that causes an executable program to remain on the swap area of the disk. It takes less time to load a program that has its sticky bit set than one that does not. Only Superuser can set the sticky bit. If the sticky bit is set on a directory that is publicly writable, only the owner of a file in that directory can remove the file.

streaming tape

A tape that moves at a constant speed past the read/write heads rather than speeding up and slowing down, which can slow the process of writing to or reading from the tape. A proper blocking factor helps ensure that the tape device will be kept streaming.

streams

See connection-oriented protocol on page 927.

string

A sequence of characters.

stylesheet

See CSS on page 928.

subdirectory

A directory that is located within another directory. Every directory except the root directory is a subdirectory.

subnet

Subnetwork. A portion of a network, which may be a physically independent network segment, that shares a network address with other portions of the network and is distinguished by a subnet number. A subnet is to a network as a network is to an internet.FOLDOC

subnet address

The subnet portion of an IP address. In a subnetted network, the host portion of an IP address is split into a subnet portion and a host portion using a subnet mask (also address mask). See also subnet number.

subnet mask

A bit mask used to identify which bits in an IP address correspond to the network address and subnet portions of the address. Called a subnet mask because the network portion of the address is determined by the number of bits that are set in the mask. The subnet mask has ones in positions corresponding to the network and subnet numbers and zeros in the host number positions. Also address mask.

subnet number

The subnet portion of an IP address. In a subnetted network, the host portion of an IP address is split into a subnet portion and a host portion using a subnet mask (also address mask). See also subnet address.

subpixel hinting

Similar to anti-aliasing (page 921) but takes advantage of colors to do the anti-aliasing. Particularly useful on LCD screens.

subroutine

See procedure on page 948.

subshell

A shell that is forked as a duplicate of its parent shell. When you run an executable file that contains a shell script by using its filename on the command line, the shell forks a subshell to run the script. Also, commands surrounded with parentheses are run in a subshell.

superblock

A block that contains control information for a filesystem. The superblock contains housekeeping information, such as the number of inodes in the filesystem and free list information.

superserver

The extended Internet services daemon.

Superuser

A privileged user having access to anything any other system user has access to and more. The system administrator must be able to become Superuser to establish new accounts, change passwords, and perform other administrative tasks. The login name of Superuser is usually root. Also root or root user.

swap

The operating system moving a process from main memory to a disk, or vice versa. Swapping a process to the disk allows another process to begin or continue execution.

swap space

An area of a disk (that is, a swap file) used to store the portion of a process's memory that has been paged out. Under a virtual memory system, the amount of swap spacerather than the amount of physical memorydetermines the maximum size of a single process and the maximum total size of all active processes. Also swap area or swapping area.FOLDOC

switch

See network switch on page 945.

symbolic link

A directory entry that points to the pathname of another file. In most cases a symbolic link to a file can be used in the same ways a hard link can be used. Unlike a hard link, a symbolic link can span filesystems and can connect to a directory.

system administrator

The person responsible for the upkeep of the system. The system administrator has the ability to log in as Superuser. See Superuser on page 956.

system console

The main system terminal, usually directly connected to the computer and the one that receives system error messages. Also console and console terminal.

system mode

The designation for the state of the system while it is doing system work. Some examples are making system calls, running NFS, processing network traffic, and performing kernel operations on behalf of the system. Contrast with user mode (page 960).

System V

One of the two major versions of the UNIX system.

TC Shell

tcsh. An enhanced but completely compatible version of the BSD UNIX C shell, csh.

TCP

Transmission Control Protocol. The most common transport layer protocol used on the Internet. This connection-oriented protocol is built on top of IP (page 938) and is nearly always seen in the combination TCP/IP (TCP over IP). TCP adds reliable communication, sequencing, and flow control and provides full-duplex, process-to-process connections. UDP (page 959), although connectionless, is the other protocol that runs on top of IP.FOLDOC

tera-

In the binary system, the prefix tera- multiplies by 240 (1,099,511,627,776). Terabyte is a common use of this prefix. Abbreviated as T. See also large number on page 940.

termcap

Terminal capability. The /etc/termcap file contained a list of various types of terminals and their characteristics. System V (page 957) replaced the function of this file with the terminfo system. Mac OS X uses the terminfo system as well.

terminal

Differentiated from a workstation (page 962) by its lack of intelligence, a terminal connects to a computer that runs Mac OS X. A workstation runs Mac OS X on itself.

terminfo

Terminal information. The /usr/share/terminfo directory contains many subdirectories, each containing several files. Each of those files is named for and holds a summary of the functional characteristics of a particular terminal. Visually oriented text-based programs, such as vim, use these files. An alternative to the termcap file.

text box

In a GUI, a box you can type in.

theme

Defined as an implicit or recurrent idea, theme is used in a GUI to describe a look that is consistent for all elements of a desktop. Go to themes.freshmeat.net for examples.

thicknet

A type of coaxial cable (thick) used for an Ethernet network. Devices are attached to thicknet by tapping the cable at fixed points.

thinnet

A type of coaxial cable (thin) used for an Ethernet network. Thinnet cable is smaller in diameter and more flexible than thicknet cable. Each device is typically attached to two separate cable segments by using a T-shaped connector; one segment leads to the device ahead of it on the network and one to the device that follows it.

thread-safe

See reentrant on page 950.

thumb

The movable button in the scrollbar that positions the image in the window. The size of the thumb reflects the amount of information in the buffer. Also bubble.

TIFF

Tagged Image File Format. A file format used for still-image bitmaps, stored in tagged fields. Application programs can use the tags to accept or ignore fields, depending on their capabilities.FOLDOC

tiled windows

An arrangement of windows such that no window overlaps another. The opposite of cascading windows (page 925).

time to live

See TTL.

toggle

To switch between one of two positions. For example, the ftp glob command toggles the glob feature: Give the command once, and it turns the feature on or off; give the command again, and it sets the feature back to its original state.

token

A basic, grammatically indivisible unit of a language, such as a keyword, operator, or identifier.FOLDOC

token ring

A type of LAN (page 940) in which computers are attached to a ring of cable. A token packet circulates continuously around the ring. A computer can transmit information only when it holds the token.

tooltip

A minicontext help system that you activate by allowing your mouse pointer to hover (page 935) over a button, icon, or applet (such as those on a panel).

transient window

A dialog or other window that is displayed for only a short time.

Transmission Control Protocol

See TCP on (page 957).

Trojan horse

A program that does something destructive or disruptive to your system. Its action is not documented, and the system administrator would not approve of it if she were aware of it.

The term Trojan horse was coined by MIT-hacker-turned-NSA-spook Dan Edwards. It refers to a malicious security-breaking program that is disguised as something benign, such as a directory lister, archive utility, game, or (in one notorious 1990 case on the Mac) a program to find and destroy viruses. Similar to back door (page 922).FOLDOC

TTL

Time to live.

1. All DNS records specify how long they are good forusually up to a week at most. This time is called the record's time to live. When a DNS server or an application stores this record in cache (page 925), it decrements the TTL value and removes the record from cache when the value reaches zero. A DNS server passes a cached record to another server with the current (decremented) TTL guaranteeing the proper TTL, no matter how many servers the record passes through.

2. In the IP header, a field that indicates how many more hops the packet should be allowed to make before being discarded or returned.

TTY

Teletypewriter. The terminal device that UNIX was first run from. Today TTY refers to the screen (or window, in the case of a terminal emulator), keyboard, and mouse that are connected to a computer. This term appears in UNIX, and Mac OS X has kept the term for the sake of consistency and tradition.

tunneling

Encapsulation of protocol A within packets carried by protocol B, such that A treats B as though it were a data link layer. Tunneling is used to transfer data between administrative domains that use a protocol not supported by the internet connecting those domains. It can also be used to encrypt data sent over a public internet, as when you use ssh to tunnel a protocol over the Internet.FOLDOC See also VPN (page 960).

UDP

User Datagram Protocol. The Internet standard transport layer protocol that provides simple but unreliable datagram services. UDP is a connectionless protocol (page 927) that, like TCP (page 957), is layered on top of IP (page 938).

Unlike TCP, UDP neither guarantees delivery nor requires a connection. As a result it is lightweight and efficient, but the application program must handle all error processing and retransmission. UDP is often used for sending time-sensitive data that is not particularly sensitive to minor loss, such as audio and video data.FOLDOC

UFS

UNIX File System. The Berkeley UNIX standard filesystem, available on Mac OS X, but not used in a standard installation. Contrast with HFS+ (page 935).

UID

User ID. A number that the passwd database associates with a login name.

undecillion

In the American system, 1036. In the British system, this number is named sexillion. See also large number (page 940).

unicast

A packet sent from one host to another host. Unicast means one source and one destination.

Unicode

A character encoding that gives a unique value to every character in every language.

unmanaged window

See ignored window on page 937.

URI

Universal Resource Identifier. The generic set of all names and addresses that are short strings referring to objects (typically on the Internet). The most common kinds of URIs are URLs.FOLDOC

URL

Uniform (was Universal) Resource Locator. A standard way of specifying the location of an object, typically a Web page, on the Internet. URLs are a subset of URIs.

usage message

A message displayed by a command when you call the command using incorrect command line arguments.

User Datagram Protocol

See UDP.

User ID

See UID.

user interface

See interface on page 937.

user mode

The designation for the state of the system while it is doing user work, such as running a user program (but not the system calls made by the program). Contrast with system mode (page 957).

username

The name you enter in response to the login: prompt. Other users use your username when they send you mail or write to you. Each username has a corresponding user ID, which is the numeric identifier for the user. Both the username and the user ID are stored in the passwd database. Also login name.

userspace

The part of memory (RAM) where applications reside. Code running in userspace cannot access hardware directly and cannot access memory allocated to other applications. Also userland. See the KernelAnalysis-HOWTO.

UTC

Coordinated Universal Time. UTC is the equivalent to the mean solar time at the prime meridian (0 degrees longitude). Also called Zulu time (Z stands for longitude zero) and GMT (Greenwich Mean Time).

UTF-8

An encoding that allows Unicode (page 959) characters to be represented using sequences of 8-bit bytes.

utility

A program included as a standard part of Mac OS X. You typically invoke a utility either by giving a command in response to a shell prompt or by calling it from within a shell script. Utilities are often referred to as commands. Contrast with builtin (command) (page 924).

variable

A name and an associated value. The shell allows you to create variables and use them in shell scripts. Also, the shell inherits several variables when it is invoked, and it maintains those and other variables while it is running. Some shell variables establish characteristics of the shell environment; others have values that reflect different aspects of your ongoing interaction with the shell.

viewport

Same as workspace (page 961).

virtual console

Additional consoles, or displays, that you can view on the system, or physical, console.

virus

A crackers (page 928) program that searches out other programs and "infects" them by embedding a copy of itself in them, so that they become Trojan horses (page 958). When these programs are executed, the embedded virus is executed as well, propagating the "infection," usually without the user's knowledge. By analogy with biological viruses.FOLDOC

VLAN

Virtual LAN. A logical grouping of two or more nodes that are not necessarily on the same physical network segment but that share the same network number. A VLAN is often associated with switched Ethernet.FOLDOC

VPN

Virtual Private Network. A private network that exists on a public network, such as the Internet. A VPN is a less expensive substitute for company-owned/leased lines and uses encryption to ensure privacy. A nice side effect is that you can send non-Internet protocols, such as Appletalk, IPX, or NetBIOS, over the VPN connection by tunneling (page 959) them through the VPN IP stream.

W2K

Windows 2000 Professional or Server.

W3C

World Wide Web Consortium (www.w3.org).

WAN

Wide area network. A network that interconnects LANs (page 940) and MANs (page 942), spanning a large geographic area (typically states or countries).

WAP

A bridge or router between wired and wireless networks. Wireless access points typically support some form of access control to prevent unauthorized clients from connecting to the network. Also WAP.

Web ring

A collection of Web sites that provide information on a single topic or group of related topics. Each home page that is part of the Web ring has a series of links that let you go from site to site.

whitespace

A collective name for SPACEs and/or TABs and occasionally NEWLINEs. Also white space.

wide area network

See WAN.

widget

The basic objects of a graphical user interface. Buttons, text fields, and scrollbars are examples of widgets.

wild card

See metacharacter on page 942.

Wi-Fi

Wireless Fidelity. A generic term that refers to any type of 802.11 (page 920) wireless network.

window

On a display screen, a region that runs or is controlled by a particular program.

window manager

A program that controls how windows appear on a display screen and how you manipulate them.

Windows share

See share on page 953.

WINS

Windows Internet Naming Service. The service responsible for mapping NetBIOS names to IP addresses. WINS has the same relationship to NetBIOS names that DNS has to Internet domain names.

WINS server

The program responsible for handling WINS requests. This program caches name information about hosts on a local network and resolves them to IP addresses.

wireless access point

See WAP.

word

A sequence of one or more nonblank characters separated from other words by TABs, SPACEs, or NEWLINEs. Used to refer to individual command line arguments. In vim, a word is similar to a word in the English languagea string of one or more characters bounded by a punctuation mark, a numeral, a TAB, a SPACE, or a NEWLINE.

Work buffer

A location where vim stores text while it is being edited. The information in the Work buffer is not written to the file on the disk until you give the editor a command to write it.

working directory

The directory that you are associated with at any given time. The relative pathnames you use are relative to the working directory. Also current directory.

workspace

A subdivision of a desktop (page 929) that occupies the entire display.

workstation

A small computer, typically designed to fit in an office and be used by one person and usually equipped with a bit-mapped graphical display, keyboard, and mouse. Differentiated from a terminal (page 957) by its intelligence. A workstation runs Mac OS X on itself while a terminal connects to a computer that runs Mac OS X.

worm

A program that propagates itself over a network, reproducing itself as it goes. Today the term has negative connotations, as it is assumed that only crackers (page 928) write worms. Compare to virus (page 960) and Trojan horse (page 958). From Tapeworm in John Brunner's novel, The Shockwave Rider, Ballantine Books, 1990 (via XEROX PARC).FOLDOC

WYSIWYG

What You See Is What You Get. A graphical application, such as a word processor, whose display is similar to its printed output.

X terminal

A graphics terminal designed to run the X Window System.

X Window System

A design and set of tools for writing flexible, portable windowing applications, created jointly by researchers at MIT and several leading computer manufacturers.

XDMCP

X Display Manager Control Protocol. XDMCP allows the login server to accept requests from network displays. XDMCP is built into many X terminals.

xDSL

Different types of DSL (page 931) are identified by a prefixfor example, ADSL, HDSL, SDSL, and VDSL.

XML

Extensible Markup Language. A universal format for structured documents and data on the Web. Developed by W3C (page 960), XML is a pared-down version of SGML.

See www.w3.org/XML and www.w3.org/XML/1999/XML-in-10-points.

XSM

X Session Manager. This program allows you to create a session that includes certain applications. While the session is running, you can perform a checkpoint (saves the application state) or a shutdown (saves the state and exits from the session). When you log back in, you can load your session so that everything in your session is running just as it was when you logged off.

Z Shell

zsh. A shell (page 953) that incorporates many of the features of the Bourne Again Shell (page 924), Korn Shell (page 940), and TC Shell (page 957), as well as many original features.

Zulu time

See UTC on page 960.


Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]


SYMBOL

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

! Boolean operator 2nd 3rd 4th
! variable
!! to reexecute the previous event
!$ last word of the previous event
# comment 2nd
# prompt
# variable
#! to choose a script shell
#define C preprocessor directive
#include C preprocessor directive
$ in regular expressions
$ in variable name
$! variable
$# variable 2nd 3rd
$#argv variable
$$ variable 2nd 3rd 4th
$(...). [See Command; substitution]
$* variable 2nd
$, use with variables
$0 variable
$< variable
$? variable 2nd
$@ variable 2nd
${}, expand variable
% job & builtin
% job builtin
% job number 2nd
& background 2nd 3rd 4th 5th 6th
& background process
& bitwise operator 2nd
& Boolean operator
& in replacement string 2nd
&& Boolean operator 2nd 3rd 4th 5th 6th 7th
((...)). [See Arithmetic; evaluation]
() command grouping
() in shell functions
* in regular expressions
* special character
*/C comment
+ in full regular expressions
. (dot) builtin 2nd 3rd 4th
. directory 2nd
. in regular expressions
.. directory 2nd
..namedfork
./ to execute a file in the working directory 2nd 3rd
.a filename extension 2nd 3rd
.app filename extension
.bash_history file
.bash_login file
.bash_logout file
.bash_profile file 2nd 3rd
.bashrc file 2nd 3rd
.bmp filename extension
.bz2 filename extension 2nd 3rd
.C filename extension 2nd
.c filename extension 2nd 3rd 4th 5th
.c++ filename extension
.cc filename extension 2nd
.conf filename extension
.cpp filename extension
.cshrc file 2nd
.cxx filename extension 2nd
.dmg filename extension 2nd
.emacs file 2nd 3rd 4th
.gif filename extension
.gz filename extension 2nd
.h filename extension 2nd
.history file 2nd
.html filename extension
.i filename extension 2nd
.ii filename extension 2nd
.inputrc file
.jpeg filename extension 2nd
.jpg filename extension 2nd
.l filename extension
.local filename extension
.login file 2nd 3rd
.logout file 2nd
.m filename extension
.netrc file
.nofinger file
.o filename extension 2nd 3rd 4th 5th
.pdf filename extension
.pkg filename extension
.plan file 2nd
.plist filename extension
.profile file 2nd
.project file 2nd
.ps filename extension
.rhosts file 2nd 3rd 4th
.S filename extension
.s filename extension 2nd 3rd 4th
.sh filename extension
.so filename extension 2nd
.tar.bz2 filename extension 2nd
.tar.gz filename extension 2nd
.tar.Z filename extension 2nd
.tbz filename extension
.tbz2 filename extension
.tcshrc file 2nd
.tgz filename extension
.tif filename extension
.tiff filename extension 2nd
.txt filename extension 2nd
.tz filename extension
.vimrc file 2nd
.y filename extension
.Z filename extension 2nd
/ directory (root) 2nd
/* C comment
/Applications directory
/bin directory
/cores directory
/dev, 2nd
/dev,
kmem file
/dev,
null file 2nd 3rd 4th
random file
special files
tty file
zero file
/etc
bashrc file 2nd
crontab file
csh.cshrc file
csh.login file
csh.logout file
cshrc file
daily file
daily.local file
group file 2nd 3rd
hostconfig file
hosts file 2nd
hosts.allow file
hosts.deny file
hosts.equiv file 2nd 3rd 4th
inittab file
login file
lookupd file
mach_init.d file
monthly file
motd file 2nd
nologin.txt file
ntp.conf file
pam.conf file
pam.d directory 2nd 3rd 4th
passwd file
printcap file 2nd
profile file 2nd 3rd 4th
protocols file
rc file
rc.local file
rc.netboot file
rpc file 2nd
securetty file
services file 2nd 3rd
shells file
sudoers file
syslog.conf file
termcap file
weekly file
xinetd.conf file
/Library directory
/private
etc directory
tmp directory
var directory
/sbin directory
/System directory
/System/Library directory
/tmp directory 2nd 3rd
/Users directory
/usr
bin
include
C preprocessor
lib 2nd
cc
shared libraries
sbin
daemons
share
doc directory
file: magic file 2nd
info directory
terminfo directory
X11R6 2nd
/var
at
jobs directory
spool directory
cron
allow file
deny file
tabs file
log 2nd
lastlog file
secure.log file 2nd
system.log file 2nd 3rd
wtmp file
mail directory
root directory
spool directory 2nd
/Volumes directory
a Boolean operator 2nd 3rd
o Boolean operator 2nd
or Boolean operator
' ...'. [See Command; substitution]
100BaseT cable
10Base2 cable
10Base5 cable
10BaseT cable
169.254 address
1> redirect standard output
2> redirect standard error
802.11
: (null) builtin 2nd 3rd 4th
:= assign default value
:= substitute default value
:? display error message
; command separator
< redirect standard input 2nd 3rd
<& duplicate input file descriptor
<0 redirect standard input
<< here document 2nd
> redirect standard output 2nd 3rd 4th 5th 6th
>& duplicate output file descriptor 2nd
>& redirect standard output and standard error
>> append standard output 2nd 3rd
? in full regular expressions
? special character
@ builtin 2nd 3rd 4th
@ in a network address 2nd
@ variable
@ vim
@ with mail
[[...]] builtin
[[...]]. [See also Conditional expressions]
[] builtin. [See test]
[] character class 2nd 3rd 4th
\ escape character 2nd 3rd
\( in regular expressions
\) in regular expressions
\n in replacement strings
^ bitwise operator
^ in regular expressions
^ quick substitution character
{ expansion
{ in a shell function
| bitwise operator
| Boolean operator 2nd
| in full regular expressions
| pipe 2nd 3rd
|| Boolean operator 2nd
|| Boolean operator 2nd 3rd 4th
} expansion
} in a shell function
~ (tilde) expansion 2nd
~ home directory
~ home directory. [See also Home directory]


A

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

a filename extension 2nd 3rd
a.out file 2nd
Abort execution
Absolute pathname 2nd 3rd 4th
Access
Access Control List. [See ACL]
Access permission
about
change using chmod
chmod
defined 2nd
description of
directory
display using ls
execute 2nd
group
other
owner
read
write
ACL 2nd 3rd 4th 5th
Active window
add command (cvs)
addbanner shell script
Address
IP
class
representation
MAC
mask 2nd
network 2nd
space, private
admin group
Administrator, system. [See System; administrator]
AIM
Airport
Algorithm
Alias
bash
defined
double versus single quotation marks
quotation marks
recursion
recursive plunge
single versus double quotation marks
special
alias builtin 2nd 3rd
alloc builtin
allow file
Alphanumeric character
ALT key
Ambiguous file reference 2nd
American National Standards Institute
AND bitwise operator 2nd
AND Boolean operator 2nd
Andreessen, Marc
Angle bracket
Animate
Anonymous FTP 2nd
ANSI
ANSI C
ansi terminal name
Answers to questions, finding
Anti-aliasing 2nd
API 2nd
app filename extension
Append
defined
standard output 2nd 3rd
Applet, defined
AppleTalk
Application Program Interface. [See API]
Applications directory
apropos utility 2nd 3rd 4th
Archive
defined
library, using
pack using tar
shell
unpack using tar
Argument 2nd
command line
defined
display
testing
argv variable
Arithmetic
bash
expansion
expression
Arithmetic evaluation
about
example 2nd 3rd
Array
argv
defined
numeric variables
string variables
Arrow keys
ASCII
codes and emacs
defined
EBCDIC conversion
man page
terminal
ascii file
ASP
Assembly language 2nd
Assignment statement, tcsh
Associative array
Asterisk special character
Asynchronous communication
Asynchronous event
at utility
AT&T Bell Laboratories 2nd 3rd 4th
ATM link
atq utility
atrm utility
atrun daemon
Attachment, defined
Attributes
extended
file
Authentication
defined
OpenSSH
user
autoconf utility
autocorrect variable
autolist variable 2nd
autologout variable
Avoided, defined
awk 2nd 3rd
format
operator 2nd
printf function
sprintf function


B

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

B language
Back door
Back tick 2nd 3rd 4th
Background
command grouping
defined
foreground, versus
job control 2nd
PID stored in $!
process 2nd
running a command in the
symbol (&)
BACKSLASH escape character 2nd 3rd
BACKSLASH in replacement strings
BACKSPACE key, changing
badtabs.c program
Basename
basename utility 2nd 3rd
bash
x option
<& duplicate input file descriptor
>& duplicate output file descriptor
alias
arguments
arithmetic
arithmetic evaluation
about
example 2nd 3rd
operators
arithmetic expansion
operators
array variables
attribute
array
export
function
integer 2nd
readonly 2nd
background
builtin [See also Builtin]
exec
getopts
kill
typeset
close file
command
process
substitution
command line, order of expansion
conditional expression
about
example 2nd
control structure. [See Control; structure]
defined
directory stack manipulation
editing previous commands 2nd
emacs command-line editor
event number
expand null variable
expand unset variable
expression
features
file descriptor
globbing
history mechanism 2nd
makepath shell script
menu
open file
operator
about
bitwise
remainder
short-circuiting
ternary
options. [See bash; features]
overlay
pathname completion
process substitution
program structures
programming
prompt
PS3 prompt
quick substitution
quiz shell script
quotation mark removal
recursion
redirection operators
reexecuting events 2nd
REPLY keyword variable
signal names 2nd
special characters
standard error
standard input
standard output
startup files
string pattern matching
substitution, quick
symbolic link
tcsh, features shared with
ternary operator
tilde substitution
variable [See also Variable]
array
assign default value
COLUMNS
display error message
expansion
LINES
modifier
OPTARG
OPTIND
PS3
REPLY 2nd
substitute default value
vi command line editor
BASH_ENV variable
bashrc file
batch utility
Baud
Baud rate
BCPL language
Bell Laboratories. [See AT&T Bell Laboratories]
Berkeley
Internet Name Domain. [See DNS]
UNIX 2nd 3rd
Berners-Lee, Tim
bg builtin 2nd 3rd 4th
bin directory
Binary file
Binary files, fixing broken
BIND
bind builtin
Binding, key
bindkey builtin
birthday shell script
bison utility
Bit
bucket 2nd
depth
Bit-mapped display
Bitwise operator
& 2nd
^
AND 2nd
|
Blank character 2nd 3rd 4th
Block
defined
device
number
special file
Blocking factor
bmp filename extension
Boolean operator
! 2nd 3rd 4th
&
&& 2nd 3rd 4th 5th 6th 7th
a 2nd 3rd
o 2nd
or
about
NOT
SPACE 2nd
| 2nd
| |
|| 2nd 3rd 4th 5th
Boot
bootstrap
defined
loader
netboot
system
verbose
Bootstrap
Bottleneck, network
Bourne Again Shell. [See bash]
Bourne Shell (original) 2nd
Bourne, Steve 2nd
Brace
around a variable
defined
expansion
shell functions
Bracket
character class
filename expansion
Branch
break control structure 2nd
break shell keyword
breaksw shell keyword
Bridge, network
Bringing the system down
Broadcast
about
address
defined
network
packet 2nd
unicast, versus
Browser
defined
Mosaic
Mozilla 2nd
Safari 2nd 3rd
BSD. [See Berkeley; UNIX]
Buffer
defined
disk
Numbered, vim
Work, vim
Builtin
% job
% job &
. (dot) 2nd 3rd
: (null) 2nd 3rd 4th
@ 2nd 3rd 4th
[[...]]
[]. [See test]
about 2nd 3rd 4th
alias 2nd 3rd
alloc
bash, list of
bg 2nd 3rd 4th
bind
bindkey
builtins
cd 2nd 3rd 4th 5th 6th 7th 8th
chdir
command
command editing using fc
declare 2nd
defined 2nd
dirs 2nd 3rd 4th
echo 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th
eval 2nd 3rd
exec 2nd 3rd 4th 5th 6th
execution of
exit 2nd 3rd 4th 5th 6th 7th
export 2nd 3rd 4th 5th
fc
fg 2nd 3rd 4th
filetest 2nd
getopts 2nd
glob
hashstat
history 2nd 3rd 4th
jobs 2nd 3rd 4th 5th 6th
kill 2nd 3rd 4th 5th 6th 7th 8th 9th 10th
let 2nd
limit
locale
log
login
logout 2nd 3rd
lsF 2nd
nice 2nd
nohup
notify
null 2nd
onintr 2nd
popd 2nd 3rd
printenv
pushd 2nd 3rd
pwd 2nd 3rd
read 2nd 3rd 4th 5th
readonly 2nd 3rd 4th
rehash
repeat
sched
set 2nd 3rd 4th 5th 6th 7th 8th
setenv 2nd 3rd
shift 2nd 3rd 4th
source 2nd
stop
suspend
tcsh
test 2nd 3rd 4th 5th 6th 7th 8th 9th
time 2nd
times
tput
trap 2nd 3rd
type 2nd
typeset
ulimit
umask 2nd 3rd
unalias 2nd 3rd 4th
unhash
unlimit
unset 2nd 3rd 4th 5th 6th
unsetenv 2nd
utility, versus
wait 2nd
where
which 2nd
builtins builtin
Bundle identifier
bundle shell script
bunzip2 utility 2nd
Byte, defined
bz2 filename extension 2nd 3rd
bzcat utility 2nd
bzip2 utility 2nd 3rd
bzip2recover utility 2nd


C

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

C filename extension 2nd
c filename extension 2nd 3rd 4th 5th
C programming language
#include preprocessor directive
a.out file
about 2nd
archived library, about
assembler
badtabs.c program
comments
compiler
phase
using
warning options
debugging
defined
expression
function prototype
functions
getchar macro
header file 2nd
include file 2nd
library
about
libc.dylib
libm.dylib
link editor
macro expansion
main function
object file 2nd
operator
optimizer 2nd
portability
preprocessor 2nd
preprocessor directives
programming, about
putchar macro
sample program
shared library, about
statically linked library, about
stdio.h header file
symbolic debugger
tabs.c program
c++ filename extension
C++ programming language
C89 programming language
Cable
100BaseT
10Base2
10Base5
10BaseT
Category 5
Category 5e
Category 6 2nd
Category 7
coaxial
fiberoptic
glass
thicknet
thinnet
twisted pair
unshielded twisted pair
UTP
Cache
cal utility
Call by value
Calling environment
Calling program, name of
Cambridge culture
Caret in regular expressions
Carnegie Mellon University. [See CMU]
Cascading windows
case control structure 2nd
Case preserving filesystem
Case sensitive
defined
domain name, not
filename
cat utility 2nd 3rd 4th 5th 6th 7th
Category 5 cable
Category 5e cable
Category 6 cable 2nd
Category 7 cable
Catenate 2nd 3rd 4th
cc filename extension 2nd
cc. [See gcc or C programming language]
cd builtin 2nd 3rd 4th 5th 6th 7th 8th
CDPATH variable 2nd
cdpath variable 2nd 3rd
CERN
Chain loading
Change
access permission using chmod
directories using cd
filename using mv
password using passwd
password using System Preferences
Character
alphanumeric
class 2nd 3rd
device
escaping
list. [See Character; class]
quoting a
special file
typeface
Character-based
defined
terminal
chdir builtin
checkout command (cvs) 2nd
Checksum
Cheetah
chflags utility
chgrp utility
Child
directory 2nd
process 2nd 3rd
chkargs shell script 2nd
chmod utility 2nd 3rd 4th
chmod() system call
chown utility
chown() system call
chroot jail
running a service in
security considerations
setting up
using
chsh utility 2nd
CIDR 2nd
CIFS
CIPE
Cipher
Ciphertext
Clark, Jim
Class, IP address
Classless Inter-Domain Routing. [See CIDR]
cleanup shell script
Clear screen
clear utility
Cleartext
CLI
Client
specifying
Client/server model 2nd 3rd
Clones 2nd
Close() files, bash
close() system call
cmp utility
CMU
Coaxial cable
Code, reentrant
CODEC
Collating sequence, machine
Colon (:) builtin
Color depth
Color quality
COLUMNS variable
Combo box
comm utility
Command
; separator
argument
builtin 2nd 3rd
completion
control flow. [See Control; structure]
control structure. [See Control; structure]
defined 2nd
editing previous
execution of
grouping 2nd
interpreter
mode, vim
name
NEWLINE separator
option
process
reexecuting previous
repeating
separation
separator 2nd
substitution 2nd 3rd 4th 5th
summary
syntax
terminator
usage message 2nd
Command line
about
argument 2nd
defined 2nd
editing
execution
expansion 2nd
interface
option
parse 2nd
processing 2nd
syntax
token 2nd
whitespace
word 2nd
COMMAND-V key
command_menu shell script
Comments
C programs
makefile
shell scripts
commit command (cvs)
Common UNIX Printing System. [See CUPS]
Communication
interprocess 2nd
network
write
comp.lang.c newsgroup
comp.os.unix.misc newsgroup
comp.sys.mac.announce newsgroup
comp.sys.mac.general newsgroup
comp.sys.mac.system newsgroup 2nd 3rd 4th
Compare files using diff
Compiler
g++
gcc
Compiling a C program
Completion
command
pathname
Readline Library
variable
Completion. [See also Word; completion]
Component architecture
Compress
bunzip2
bzcat
bzip2 2nd 3rd
bzip2recover 2nd
compress 2nd 3rd 4th
gunzip
gzip 2nd 3rd
uncompress
unzip
zcat 2nd
zip
compress utility 2nd 3rd 4th
Computer Systems Research Group
Computer, diskless
Computing, distributed 2nd
Concatenate. [See Catenate]
Concurrent Versions System. [See cvs command]
Condition code. [See Exit; status]
Conditional expression
about
example 2nd
conf filename extension
Configuration file. [See Startup file]
configure utility
Connection-oriented protocol 2nd
Connectionless, protocol
Console
single-user mode
terminal
text 2nd
virtual
Context, diff
continue control structure 2nd
continue shell keyword
Control
character
characters, printer
continue 2nd
defined
do 2nd 3rd 4th
done 2nd 3rd 4th
elif
elif vs. fi
else
esac
fi 2nd 3rd
fi vs. elif
flow. [See Control; structure]
for 2nd 3rd
for...in 2nd 3rd
foreach
goto
if 2nd 3rd 4th 5th
if...then
if...then...elif 2nd
if...then...else 2nd
in
job
select
shell keyword
break
breaksw
case
continue
default
else
endif
endsw
switch
then
while
shell scripts
structure
about 2nd
break 2nd
case 2nd
switch
then 2nd 3rd
two-way branch
until
while 2nd 3rd 4th
CONTROL key
CONTROL-C key 2nd
CONTROL-D key 2nd 3rd 4th
CONTROL-H key 2nd 3rd
CONTROL-L key 2nd 3rd
CONTROL-M key
CONTROL-R key
CONTROL-U key 2nd 3rd 4th
CONTROL-V key
CONTROL-W key 2nd 3rd
CONTROL-X key
CONTROL-Z key 2nd
Convention
book, used in this
end line key
file naming
Cookie
Coordinated Universal Time. [See UTC]
Copy
directory recursively using mv
directory, shell script
file using cp 2nd
Core
dump
file
memory
Correct
typing mistakes
vim typing mistakes
correct variable
count shell script
cp utility 2nd 3rd 4th
cp vs. ln
cpdir shell script
cpio utility
cpp filename extension
Cracker
Crash 2nd
creat() system call
Create
directory using mkdir
file using vim
file, null
Creation date, file, display using ls
creator code
cron daemon 2nd
cron directory
cron utility 2nd
cron, Vixie
crontab file 2nd
crontab utility
Cryptography
csh 2nd
csh.cshrc file
csh.login file
csh.logout file
cshrc file
CSRG
CSS 2nd
CUPS, about
Current
Current directory. [See Working directory]
Cursor
cut utility 2nd 3rd
cvs command
add
checkout 2nd
commit
export
import
log
release
remove
rtag
update
cvs utility
CVSROOT variable
cwd variable
cxx filename extension 2nd
Cypher. [See Cipher]


D

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

Daemon
atrun
cron 2nd
defined
fingerd 2nd 3rd
ftpd
httpd
init 2nd
launchd 2nd 3rd 4th 5th 6th 7th
lpd
mach_init
name
naming
network 2nd
nmbd
ntpd
portmap 2nd 3rd
postfix
protocol
rexecd
rpc. prefix
services, providing
smbd
sshd 2nd
Superserver. [See Daemon; xinetd]
syslogd 2nd
talkd
telnetd
xinetd 2nd 3rd 4th
daily file
daily.local file
Darwin
about
version numbers
Data
sink
structure
Database
dbm
DNS
gdbm
hosts
locale
magic number
man page header
ndbm
NIS 2nd
printcap
search engine
services
SQL
system services
terminfo
whois
Datagram, network 2nd
Dataless system
date utility 2nd 3rd 4th
Date, display
dbm database
dd utility
ddd utility
DDoS attack
Debug
Debugger
about
ddd
gdb
graphical symbolic
option
shell script
symbolic
declare builtin 2nd
Decrement operator 2nd
Default
default shell keyword
Delete
directory using rmdir
file
line
link using rm
word
DELETE key
Delimiter, regular expression
Delta, SCCS
Denial of Service. [See DoS attack and DDoS attack]
deny file
Dependency line (make)
Dereference
Descriptor, file
Descriptor, file, duplicate
Design, top-down
Desktop
defined
manager 2nd
Desktop directory
Detached process. [See Background; process]
dev directory 2nd
Device
block
character
defined
driver 2nd 3rd 4th
file 2nd
filename
independence
independent input and output
null
number
major 2nd
minor 2nd
physical
df utility
DHCP
about
client
how it works
more information
protocol
server
Die, process
diff utility 2nd
diff, context
diff3 utility
dig utility
Directory
.
..
/ (root) 2nd
/Applications
/bin
/dev
about
special files
/etc
pam.d 2nd 3rd 4th
X11
/Library
/private
etc
tmp
var
/sbin
/System
Library
/tmp 2nd 3rd
/Users
/usr
/share doc
bin
include
info
lib 2nd
lib: cc, and
lib: shared
sbin
share
share: terminfo
X11R6 2nd
/var
at
at: jobs
at: spool
log 2nd
root
spool 2nd
/Volumes
about
access permission
Applications
bin 2nd
change using cd
child 2nd
copy recursively using mv
copy, shell script
cores
create using mkdir
cron
current. [See Directory; working]
defined 2nd 3rd 4th
delete using rmdir
Desktop
dev 2nd 3rd
doc
empty
erase using rmdir
etc
file 2nd
hierarchy
home 2nd 3rd 4th
home vs. working
include 2nd
info
lib 2nd 3rd 4th
Library 2nd
link
list using ls
listing
log 2nd
lost+found
make using mkdir
move using mv
pam.d 2nd 3rd 4th
parent 2nd
pathname 2nd
private
remove using rmdir
rename using mv
root 2nd 3rd
root
root
sbin 2nd
service
share
spool 2nd 3rd
stack manipulation 2nd 3rd
standard
System
terminfo
tmp 2nd
Users
var
Volumes
working
change using cd
defined
home, versus
relative pathnames
significance of
with
X11
X11R6 2nd
~ (home). [See Home directory]
dirs builtin 2nd 3rd 4th
dirstack variable
Disk
buffer
free space
partition
startup
usage
diskutil utility
Display
date using date
end of a file with tail
file using cat
graphical
invisible filename
machine name
ordered file using sort
sorted file using sort
system load using w
text using echo
top of a file using head
uptime using w
user list using finger
user list using w
user list using who
Distributed computing 2nd
ditto utility
dmesg utility 2nd
dmg filename extension 2nd
DMZ
DNS
about 2nd
database
dig utility
host utility
do control structure 2nd 3rd 4th
doc directory
Document Object Model. [See DOM]
Documentation
finding
online
system 2nd
Dollar sign
regular expression, in
variables, use with 2nd
DOM
Domain
name
Name Service. [See DNS]
name, not case-sensitive
done control structure 2nd 3rd 4th
Door
DOS
filename
filename extension
DoS attack
Double versus single quotation marks
DPMS
Drag
Driver, device 2nd
DSA
DSL 2nd
du utility 2nd
dunique variable
Duplex, full and half
Duplicate lines, getting rid of using uniq
dyld utility 2nd
dylib filename extension 2nd
Dynamic IP address
Dynamic library. [See Shared; library]


E

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

EBCDIC
echo builtin 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th
echo utility
echo variable
ed editor 2nd 3rd
Editor
command line
defined
ed 2nd 3rd
emacs
ex 2nd 3rd
Readline Library
vim
EDITOR variable 2nd
Edwards, Dan
EEPROM
Effective user ID
egrep utility 2nd 3rd
Electronic mail. [See Mail]
Electronic message, sending using write
Element
ELF binary format
elif control structure
else control structure
else shell keyword
elvis utility
emacs
*compilation* buffer
*Tab Stops* buffer
about
acronyms
ALT key
Auto Fill mode
background shell commands
backup
buffer
bufferwide replacement
bufferwide search
current
saving and retrieving
state
working with
C mode
case conversion
comments
CONTROL key
current buffer
cursor
customizing
cut text
delete
Dired mode
Echo Area
edit commands
editing at the cursor
erase text
escape character
ESCAPE key 2nd
expressions, working with
FIFO
files
fill
foreground shell commands
function, defining
global commands
GUD mode
help
human-language modes
incremental search
indention
insert text
interactive replacement
key
notation
remap
sequences and commands
use
keymaps
keystroke notation
kill
Kill Ring 2nd 3rd
kill text
language-sensitive editing
Lisp system, internal
major modes 2nd
Mark
Mark Ring
META key 2nd
Minibuffer
Mode Line 2nd
move by
characters
lines
paragraphs
sentences
window position
words
move the cursor
multilevel backups
nonincremental search
numeric argument
online help
paragraphs, working with
paste text
Point 2nd
Region
regular expression search
remap keys
repetition count
replacement
resources
retrieving the buffer
Rmail mode
save, buffer
save, file
scrolling through a buffer
search
bufferwide
incremental
nonincremental
regular expression
types of
sentences, working with
shell commands
Shell mode
SHIFT key
smart completion
special characters
special-purpose modes
start
stop
subshell
TAB key
Text mode
textual units
Tramp mode
unconditional replacement
undo changes
VC mode
vi simulation
vi, vs.
VIP mode
visiting a file 2nd
window
adjust
create by splitting
delete
manipulate
other window
working with
words, working with
yank text
emacs utility
Email. [See Mail]
Emoticon
Empty regular expression
End line key
End of file. [See EOF]
endif shell keyword
endsw shell keyword
Enquire program
Enter text using vim
ENV variable
Environment
calling
defined
establish
exec
export 2nd 3rd
setenv
unsetenv
variable
EOF 2nd 3rd
Epoch, defined
EPROM
Erase key 2nd 3rd 4th 5th 6th 7th
Erase word key
Error
correcting
message
conditional
name of the calling script
not found
Permission denied
redirecting to standard error
standard error. [See Standard; error]
usage 2nd
shell script
standard. [See Standard; error]
usage message 2nd
esac control structure
Escape a character 2nd
Escape an end of line
etc directory
Ethernet network 2nd 3rd 4th 5th
Eumlation, terminal
eval builtin 2nd 3rd
Event
asynchronous
defined
history
modifying previous
number 2nd 3rd
reexecuting
text
words within
ex editor 2nd
Exabyte
Exclamation point
exec builtin 2nd 3rd 4th 5th 6th
exec() system call 2nd
Execute
access 2nd 3rd
command 2nd
permission
shell script
Exit
shell, from a
status 2nd
exit builtin 2nd 3rd 4th 5th 6th 7th
exit() system call
Expansion
arithmetic
brace 2nd
command line
filename
macro
null variable
order of 2nd
parameter
pathname 2nd 3rd
quotation marks, double
tilde 2nd
unset variable
variable
Exploit
Export a variable
export builtin 2nd 3rd 4th 5th
export command (cvs)
expr utility
Expression
arithmetic
defined
logical
Extended attributes
Extended regular expression 2nd
Extensible Markup Language. [See XML]
Extension, filename. [See Filename; extension]
Extranet 2nd


F

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

Fahlman, Scott
Failsafe session
Family tree
FAT filesystem. [See Filesystem; FAT]
fc builtin
FCEDIT variable
FDDI network
fg builtin 2nd 3rd 4th
fgrep utility 2nd
fi control structure 2nd 3rd
Fiber Distributed Data Interface. [See FDDI network]
Fiberoptic cable
FIFO special file 2nd 3rd 4th 5th
fignore variable
File
.bash_history
.bash_login
.bash_logout
.bash_profile 2nd 3rd
.bashrc 2nd 3rd
.cshrc 2nd
.emacs 2nd 3rd 4th
.history 2nd
.inputrc
.login 2nd 3rd
.logout 2nd
.netrc
.nofinger
.plan 2nd
.profile 2nd
.project 2nd
.rhosts 2nd 3rd 4th
.tcshrc 2nd
.vimrc 2nd
/dev
kmem
null 2nd 3rd 4th
random
tty
zero
/etc
bashrc
crontab
csh.cshrc
csh.login
csh.logout
cshrc
daily
daily.local
group 2nd
hostconfig
hosts 2nd
hosts.allow
hosts.deny
hosts.equiv 2nd 3rd 4th
login
lookupd
mach_init.d
monthly
motd 2nd
nologin
ntp.conf
pam.conf
pam.d
passwd
printcap 2nd
profile 2nd 3rd 4th
protocols
rc
rc.local
rc.netboot
rpc 2nd
services 2nd 3rd
shells
sudoers
syslog.conf
termcap
ttys 2nd
weekly
xinetd.conf
/usr
include
sbin
share
share: file/magic 2nd
share: misc/ascii
share: misc/man.conf
/var
cron
cron: allow
cron: deny
cron: tabs
log
log: lastlog
log: secure.log 2nd
log: system.log 2nd 3rd
log: wtmp
a.out 2nd
access permission 2nd
allow
ambiguous reference
archive using tar
ascii
attributes 2nd
bashrc 2nd
binary, fixing broken
block special 2nd
character special 2nd
close (bash)
configuration, rules
copy using cp
core
create using cat
creation date, display using ls
crontab 2nd
csh.cshrc
csh.login
csh.logout
cshrc
daily
daily.local
data fork
defined 2nd
deny
descriptor 2nd
descriptor, duplicate
device 2nd 3rd
directory 2nd 3rd
directory vs. ordinary
display beginning of using head
display end of using tail
display using cat
execute permission
FIFO special 2nd 3rd 4th
flags
fork
data
resource
GNUmakefile
group 2nd
group assignment
group, display using ls
header 2nd
hierarchical structure
hostconfig
hosts 2nd
hosts.allow
hosts.deny
hosts.equiv 2nd 3rd 4th
identifying using file
include
kmem
ktrace.out
lastlog
link
list
log
login
lookupd
mach_init.d
magic 2nd
Makefile 2nd 3rd
makefile 2nd
man.conf
monthly
motd 2nd
move using mv
name. [See Filename]
named pipe 2nd 3rd 4th
nologin
ntp.conf
null 2nd 3rd 4th
object 2nd
open, bash
order using sort
ordinary 2nd 3rd
ordinary vs. directory
owner
owner, display using ls
pack archive using tar
pam.conf
pam.d
passwd
pathname 2nd
permission 2nd 3rd
pointer to
printcap 2nd
profile 2nd 3rd 4th
property list 2nd
protocols
random
rc
rc.local
rc.netboot
reference, ambiguous 2nd
remove using rm
rename using mv
resource fork
rotate
rpc 2nd
screen
secure.log 2nd
services 2nd 3rd
shells
size, display using ls
sort using sort
sparse 2nd
special 2nd 3rd 4th
standard
startup 2nd 3rd 4th
stdio.h C header
structure
sudoers
swap
symbolic link
syslog.conf
system.log 2nd 3rd
tabs
tar
temporary, name of
termcap
terminal
text
truncate
tty
ttys 2nd
type code
type of, discover using ls
typescript
UNIX domain socket
weekly
wtmp
xinetd.conf
zero
file utility 2nd 3rd
filec variable
Filename
/
about 2nd
ambiguous reference. [See File; ambiguous reference]
Carbon
case
case sensitive
change using mv
characters in
choice of
completion
conventions
defined 2nd
device
DOS
extension
a 2nd 3rd
about
app
bmp
bz2 2nd 3rd
C 2nd
c 2nd 3rd 4th 5th
c++
cc 2nd
conf
cpp
creator codes, and
cxx 2nd
defined
dmg 2nd
DOS
dylib 2nd
dynamic library
emacs and
gif
gz 2nd 3rd
h 2nd
html
i 2nd
ii 2nd
jpeg 2nd
jpg 2nd
l
list of 2nd
local
m
o 2nd 3rd 4th 5th
pdf
pkg
plist
ps
remove a
S
s 2nd 3rd 4th
sh
tar.bz2 2nd
tar.gz 2nd
tar.Z 2nd
tbz
tbz2
tgz
tif
tiff 2nd
txt 2nd
tz
y
Z 2nd
generation 2nd 3rd
invisible 2nd 3rd
length 2nd
period, leading
quoting
reference, ambiguous. [See File; ambiguous reference]
root directory
simple 2nd 3rd 4th
substitution
temporary file 2nd
typeface
unique 2nd
Windows
Fileserver
Filesystem
case preserving
case sensitivity
copy directory recursively using mv
defined 2nd
FAT
filename length
free list 2nd
HFS 2nd
HFS+ 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th
HFS+ Extended. [See Filesystem; HFSX]
HFSX 2nd 3rd
journaling
non-disk
organize
repair
root
structure 2nd
UFS 2nd 3rd 4th 5th
use
filetest builtin 2nd
Filling
Filter 2nd 3rd 4th
Find
command name using apropos
string using grep
find utility 2nd
find utility, inode 2nd
find_uid shell script
finder shell script
finger utility 2nd 3rd 4th 5th 6th
fingerd daemon 2nd 3rd
fink utility
Firewall, about
firmware
Flags, file
flex utility
fmt utility
Folder. [See Directory]
Footer
for control structure 2nd 3rd
for...in control structure 2nd 3rd
foreach control structure
Foreground
background, versus
defined
process
Fork
child
data
defined
file
process
resource 2nd 3rd 4th
fork() system call 2nd 3rd 4th
FQDN 2nd 3rd
Frame, network 2nd
Frameworks
Free
list, filesystem 2nd
space
space, disk
fsaclctl utility
fsck utility 2nd
FTP
active connection
anonymous
automatic login
passive connection 2nd
PASV connection
PORT connection
pub directory
tutorial
ftp utility 2nd
ftpd daemon
Full
duplex 2nd
regular expressions
egrep
pipe
plus sign
question mark
summary
Fully qualified domain name. [See FQDN]
Function
C language 2nd
defined
prototype
shell 2nd


G

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

g++ compiler
Gateway
network
proxy
gcc [See also C programming language]
compiler
history
home page
using
warning options
gdb utility 2nd 3rd
gdbm database
GECOS
Generate filenames
getchar C macro
GetFileInfo utility
getopts builtin 2nd
getty utility 2nd
gid variable
gif filename extension
Giga-
glob builtin
Global variable 2nd
Globbing 2nd 3rd 4th
Glyph
GMT. [See UTC]
GNU
Compiler Collection
emacs
gcc
compiler 2nd
home page
gdb utility
manuals
usage message
GNUmakefile file
gopher utility
goto control structure
gprof utility
Graphical display
Grave accent. [See Back tick]
Greenwich Mean Time. [See UTC]
grep utility 2nd 3rd 4th 5th 6th 7th 8th 9th
Group
about
access
admin
changing
commands 2nd
file assigned to
groups utility
ID 2nd 3rd 4th
name of, display using ls
user private
users
wheel
windows
group file 2nd
groups utility
GUI
combo box
defined
radio button
scrollbar
spinner
text box
thumb
wysiwyg
gunzip utility 2nd
gz filename extension 2nd
gzip utility 2nd 3rd


H

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

h filename extension 2nd
Hacker
Half-duplex 2nd
halt utility 2nd
Halt, program
Hard link
create using ln 2nd
defined 2nd
number of, using ls
remove using rm
symbolic link, versus
Hardcoded filename in shell scripts
Hash
defined
one-way
table
hashstat builtin
hdiutil utility
head utility 2nd
Header
document
file 2nd
Help
answers, finding
Apple Help Viewer
Apple Web site
apropos utility
documentation
emacs
error messages
getting
GNU manuals
hardware
info pages
Internet, from the
man pages
netnews
support forums
Here document 2nd 3rd
Hesiod
Heterogeneous
Hexadecimal number
HFS+ Extended filesystem. [See Filesystem; HFSX]
HFS+ filesystem. [See Filesystem; HFS+]
HFSX filesystem. [See Filesystem; HFSX]
Hidden file. [See Invisible filename]
Hierarchical file structure
Hierarchy
histfile variable 2nd 3rd
HISTFILESIZE variable
histlit variable 2nd
History
C Shell Mechanism, classic
defined
event
editing
number
previous
previous: !$ last word of the
previous: modifying
reexecuting
text
words within
mechanism 2nd
vi, of
viewing
word designator
history builtin 2nd 3rd 4th
history variable 2nd
HISTSIZE variable
Home directory
.bash_history file
.bash_login file
.bash_logout file
.bash_profile file 2nd
.bashrc file
.cshrc file
.history file 2nd
.inputrc file
.login file 2nd 3rd
.logout file 2nd
.netrc
.nofinger file
.plan file
.profile file 2nd
.project file
.rhosts
.tcshrc file 2nd
.vimrc file 2nd
about
defined 2nd
Desktop
invisible file
Library
startup file
working directory, versus
~, shorthand for
HOME variable 2nd 3rd 4th 5th
home variable
Host
address
specifying
trusted
host utility
hostconfig file
hostname utility 2nd
Hostname, symbolic
hosts database 2nd
hosts file 2nd
hosts.allow file
hosts.deny file
hosts.equiv file 2nd 3rd 4th
Hover
HTML
history
suffix
html filename extension
HTTP 2nd
httpd daemon
HTTPS protocol
Hub 2nd
Human-readable option
Humor 2nd 3rd
hunk (diff)
Hypermedia
Hypertext
about
defined
link
Markup Language. [See HTML]
Transfer Protocol. [See HTTP]
World Wide Web


I

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

i filename extension 2nd
I/O device. [See Device]
IANA 2nd
ICMP packet 2nd
icmp_seq
Icon
Iconify
ICQ
id utility
Idd utility
if control structure 2nd 3rd 4th 5th
if...then control structure
if...then...elif control structure 2nd
if...then...else control structure 2nd
IFS variable
Ignored window
ignoreeof variable 2nd
ii filename extension 2nd
Implied dependency, make
import command, cvs
in control structure
Include directive
include directory 2nd
Include file
Incorrect login
Increment operator
indent shell script
Indentation. [See Indention]
Indention
Indirect pointer
Infinite recursion, alias
info directory
info utility
manual
using
init daemon 2nd
Init script
inittab file
Inode
altering using mv
create another reference using ln
defined 2nd 3rd
display using ls
file
filesystem
links shell script
number
Input
defined
mode, vim
standard. [See Standard input]
Input/output device. [See Device]
input_line variable
INPUTRC variable
Installation, computer
Integrated Services Digital Network. [See ISDN]
Interactive
Interface
command line
defined
user 2nd
Internal Field Separator. [See IFS variable]
International Organization for Standardization. [See ISO]
Internet
about
Assigned Numbers Authority. [See IANA]
browser
Control Message Protocol. [See ICMP]
defined
look up a user
mirror site
netiquette
netnews. [See Netnews]
network
Printing Protocol. [See IPP protocol]
Protocol. [See IP and TCP]
Relay Chat
search engine
service provider. [See ISP]
services
speed
URI
URL 2nd
Usenet
internet (small i)
Internetwork
InterNIC
Interprocess communication 2nd 3rd
Interrupt handling
Interrupt key 2nd 3rd 4th 5th
Intranet 2nd
Invisible filename
defined 2nd
display
not displayed with a ?
IP
address 2nd 3rd
class, address
defined
IPng
IPv6
multicast. [See Multicast]
Next Generation
spoofing 2nd
version 6. [See IP; IPv6]
IPC
IPP protocol
IPv6 2nd 3rd
IRC
IsF builtin 2nd
is_regfile shell script
ISDN 2nd
ISO
defined
ISO9660 filesystem 2nd
protocol model
ISP


J

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

Jaguar
Job
control 2nd
about
bg builtin
defined
fg builtin 2nd
how to use
jobs builtin 2nd
notify builtin
stop builtin
defined
launchd
number 2nd
stop foreground
jobs builtin 2nd 3rd 4th 5th 6th
Jolitz, Bill
Journaling filesystem
Joy, Bill
JPEG
jpeg filename extension 2nd
jpg filename extension 2nd
Justify


K

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

K&R 2nd
kdump utility
Kerberos
Kernel
boot
defined
device driver
display messages using dmesg
Mach
module. [See Loadable module]
programming interface
responsibilities
special files
system calls
kernelspace
Kernighan & Ritchie. [See K&R]
Key
Arrow
binding
binding, emacs
COMMAND-V
CONTROL
CONTROL-C
CONTROL-D 2nd 3rd 4th
CONTROL-H 2nd 3rd
CONTROL-L 2nd 3rd
CONTROL-M
CONTROL-R
CONTROL-U 2nd 3rd 4th
CONTROL-V
CONTROL-W 2nd 3rd
CONTROL-X
CONTROL-Z 2nd
DELETE
end line
ENTER
erase 2nd 3rd 4th 5th 6th 7th
interrupt 2nd 3rd 4th
kill 2nd 3rd 4th
line kill 2nd 3rd 4th
META 2nd 3rd
NEWLINE 2nd
notation
RETURN 2nd 3rd 4th
SPACE bar
suspend 2nd 3rd 4th 5th 6th 7th 8th
TAB
typeface
word erase 2nd
Keyboard
defined
move cursor to next line
Keychain password
Keystroke notation
Keyword
searching with apropos
variable 2nd
kill builtin 2nd 3rd 4th 5th 6th 7th 8th 9th 10th
Kill key 2nd 3rd 4th
Kill line key 2nd 3rd 4th
kill utility 2nd
kill() system call
killall utility 2nd
kilo-
kmem file
Korn Shell. [See ksh]
Korn, David 2nd
ksh
defined
history
ktrace utility
ktrace.out file


L

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

l filename extension
LAN
about 2nd
compared to intranet
defined
example
Language
procedural
sensitive editor, emacs
Large number
Last in first out stack
Last Line mode, vim
lastlog file
launchctl utility 2nd
launchd daemon 2nd 3rd 4th 5th 6th 7th 8th
LD_LIBRARY_PATH variable
LDAP
Leaf
Least privilege 2nd
Length of filename
less utility 2nd 3rd 4th 5th 6th
LESS variable
let builtin 2nd
lib directory 2nd 3rd 4th
libc.dylib library
libm.a library
libm.dylib library
Library
about
archived
dynamic. [See Library; shared]
libc.dylib
libm.dylib
shared 2nd
statically linked
Library directory 2nd
LIFO stack
Lightweight Directory Access Protocol. [See LDAP]
limit builtin
Line kill key 2nd 3rd 4th
Line Printer Daemon. [See lpd daemon]
LINES variable
Link
about
create using ln
defined 2nd
delete using rm
hard 2nd 3rd 4th 5th 6th
hard versus symbolic
hypertext
inode
number of, display using ls
point-to-point
remove using rm 2nd
soft. [See Link; symbolic]
symbolic
bash, and
create using ln 2nd
defined 2nd
display using find
mark using ls
versus hard link 2nd
symlink. [See Link; symbolic]
links shell script
lint utility
Linux
PAM. [See PAM]
Pluggable Authentication Modules. [See PAM]
Lisp programming language
List operator
List server 2nd
listjobs variable
listlinks variable
Listserv 2nd
ln utility 2nd 3rd 4th
ln versus cp
Load average 2nd
Load, system, display with w
Loadable module
Local
area network. [See LAN]
variable 2nd 3rd 4th
local filename extension
Locale
locale builtin
locale database
localhost
lock utility
locktty shell script 2nd
Log
files
in. [See Login]
out 2nd
syslogd daemon
system
log builtin
log command (cvs)
log directory 2nd
Logical
expression
operator. [See Boolean operator]
Login
defined
description of
incorrect
name
problems
procedure
prompt 2nd
remote
root 2nd
shell 2nd 3rd 4th 5th
login builtin
login file
loginsh variable
loginwindow utility
LOGNAME variable
Logout
logout builtin 2nd 3rd
lookupd file
lost+found directory
lpd daemon
lpq utility 2nd
LPR line printer system
lpr utility 2nd 3rd 4th
lprm utility 2nd
lpstat utility
ls utility 2nd 3rd 4th 5th 6th 7th 8th
lseek() system call


M

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

m filename extension
MAC address
Mac OS X
manual
newsgroup
version numbers
Mach kernel
Mach-O format
mach_init daemon
mach_init.d file
Machine
collating sequence 2nd
independence
name, display
Macro
C preprocessor 2nd
defined
expansion
make
magic file 2nd
Magic number 2nd 3rd
magic number database
Mail
about
delivery agent
list server 2nd
mailbox
MDA
network addresses
sending to remote user
spam
MAIL variable
mail variable
Mailbox
MAILCHECK variable
MAILPATH variable
MAILTO variable
main function
Maintenance. [See System; maintenance]
Major device number 2nd 3rd
make utility 2nd 3rd 4th 5th 6th 7th
Makefile file 2nd 3rd
makefile file 2nd
makefile, discussion
MAN 2nd
man page header database
man utility 2nd 3rd 4th
man.conf file
MANPATH variable
Manuals
GNU 2nd
man
reference, finding
system, about
Massachusetts Institute of Technology. [See MIT]
McCarthy, John M.
MD5 encryption
MDA
Mega-
Menu
defined
shell script
Merge
mesg utility
Message
deny with mesg
Digest 5. [See MD5 encryption]
of the day. [See motd file]
sending using write
usage 2nd 3rd 4th 5th 6th
messages file 2nd
META key 2nd 3rd
Metabit
Metacharacter 2nd
Metadata 2nd
creator code
file type code
Method of Last Resort, kill
Metropolitan area network. [See MAN]
MIME
defined
type, displayed by file
Minimize window
Minor device number 2nd 3rd
Mirror site
misc.jobs.offered newsgroup
Mistake, correct typing
MIT
MIT, lisp culture
mkdir utility 2nd 3rd
mkfifo utility
Modeless editor
Modem
cable
defined
monthly file
more utility
Mosaic Web browser
motd file 2nd
Mount
automatic
defined
point
mount utility
Mouse
pointer
pointer, hover
window manager
Mouseover
Move
directory using mkdir
file using mv
Mozilla
history of
netnews
MS Windows. [See Windows]
MTA
MUA
Multiboot specification
Multicast 2nd
Multitasking 2nd 3rd
Multithreaded program
Multiuser
about
defined
Mac OS X design
mode, initiate
Superuser
mv utility 2nd 3rd 4th


N

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

Name
command
daemons
domain. [See Domain; name]
login
server 2nd
variable 2nd
Named pipe 2nd 3rd 4th 5th
NAT
National Center for Supercomputer Applications
NBT
ndbm database
NetBIOS
Netboot
NetInfo
about 2nd
database 2nd
hosts database
passwd database
Netiquette
Netmask
Netnews
answers, finding
archive, groups.google.com,
comp.lang.c newsgroup
comp.os.unix.misc newsgroup
comp.sys.mac.announce newsgroup
comp.sys.mac.general newsgroup
comp.sys.mac.system newsgroup 2nd
group structure
groups
hierarchical structure of groups
history of
misc.jobs.offered newsgroup
Mozilla News
Netscape News
newsgroups
nn utility
readnews utility
rec.skiing newsgroup
rn utility
sci.med newsgroup
soc.singles newsgroup
structure of groups
talk.politics newsgroup
tin utility
xrn utility
xvnews utility
Netscape
history of
Navigator
netnews
netstat utility
Network
100BaseT cable
10Base2 cable
10Base5 cable
10BaseT cable
@ in an address 2nd
@ with mail
address
@ sign in
defined
mail
mask
space, private
Address Translation. [See NAT]
boot
bottleneck
broadcast
about
address
defined
multicast, compared
packet 2nd
unicast, compared
browser
Category 5 cable
Category 5e cable
Category 6 cable 2nd
Category 7 cable
class, IP address
coaxial cable
connection, test using ping
daemon 2nd
datagram 2nd
DNS
domain name
dynamic IP address
Ethernet 2nd 3rd 4th 5th
extranet 2nd
FDDI
fiberoptic cable
firewall
frame 2nd
gateway 2nd
hops
host address
hostname, FQDN
hostname, nickname
hub 2nd
ICMP packet
Information Service. [See NIS]
Internet
intranet
IP
address
address class
Next Generation
Local Area. [See LAN]
Metropolitan Area. [See MAN]
multicast 2nd
nameserver 2nd
netmask
netnews. [See Netnews]
newsgroup
number. [See Network; address]
packet 2nd 3rd
packet filtering
packet sniffer
partner net
passive hub
ping to test
point-to-point link
port forwarding
PPP protocol
private address space
privileged port
protocol
remote filesystem
resolver
route trace
router 2nd 3rd 4th
segment
services 2nd
SLIP protocol
sniff
socket
static IP address
subnet
address
defined
mask 2nd
number
specifying
switch 2nd 3rd
TCP/IP protocol
thicknet cable
thinnet cable
Time Protocol. [See NTP]
token ring
topology, shared
trace route
transfer rate
trusted hosts
tunneling 2nd
twisted pair cable
UDP 2nd
unicast 2nd
unicast vs. broadcast
unshielded twisted pair cable
user communication
utilities 2nd
UTP cable
virtual private
VPN 2nd
WAN
Wi-Fi
wide area. [See WAN]
newfs utility
newfs_hfs utility
newfs_msdos utility
NEWLINE key 2nd
News, Internet. [See Netnews]
Newsgroup
comp.lang.c
comp.os.unix.misc
comp.sys.mac.announce
comp.sys.mac.general
comp.sys.mac.system 2nd 3rd
list of
misc.jobs.offered
rec.skiing
sci.med
soc.singles
talk.politics
NIC 2nd
nice builtin 2nd
nice utility
nice() system call
Nickname, host
nidump utility 2nd 3rd
nireport utility
NIS
database
defined
domain name
nmbd daemon
nn utility
NNTP 2nd
No news is good news
nobeep variable
noclobber variable 2nd 3rd
noglob variable
nohup builtin
nohup utility 2nd
nologin file
nonomatch variable
Nonprinting character 2nd
Nonvolatile storage
Normal mode, vim. [See vim; Command mode]
NOT Boolean operator
notify builtin
notify variable
NTP
ntp.conf file
ntpd daemon
Null
(:) builtin 2nd
device
string 2nd
null file 2nd 3rd 4th
Number
block
device
major
minor
giga-
hexadecimal
job
kilo-
large
magic 2nd
mega-
octal
sexillion
tera-
undecillion
Numbered Buffers, vim
Numeric variable 2nd 3rd
Numeric variable, array
nvi utility


O

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

o filename extension 2nd 3rd 4th 5th
Object file 2nd
Octal number
od utility 2nd
OLDPWD variable
onintr builtin 2nd
Online documentation 2nd
Open Firmware
open utility
open() system call
OpenSSH
authentication
defined
initial connection to
public-key encryption
Operating system, defined
Operator
bash, about
bash, in expressions
bash, redirection
bitwise
& 2nd
^
AND 2nd
I
Boolean
! 2nd 3rd 4th
&
&& 2nd 3rd 4th 5th 6th 7th
a 2nd 3rd
o 2nd
or
about
I 2nd
I I
II 2nd 3rd 4th 5th
NOT
SPACE 2nd
decrement 2nd
increment
logical. [See Operator; Boolean]
postdecrement 2nd
postincrement 2nd
predecrement
preincrement
relational
short-circuiting
table of
OPTARG variable
Optimizer, C compiler 2nd
OPTIND variable
Option
bash. [See bash, features]
combining
defined 2nd
help
human readable
OPTION key
OR operator
Order file with sort
Order of expansion, command line
Ordinary file 2nd
Organize a filesystem
Other access
otool utility 2nd 3rd
out shell script
Output
append. [See Append; standard output]
defined
redirect
standard. [See Standard; output]
Overlay a shell
owd variable
Owner
access
file, name of, display using ls 2nd


P

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

P2P
Pack archive file using tar
Packet
broadcast
defined
filtering
network 2nd
sniffer
unicast
Page break
Pager 2nd 3rd
PAGER variable 2nd
Paging
PAM
about
features
stack
pam.conf file
pam.d directory 2nd 3rd 4th
Panes
Panther
Parameter
expansion
positional 2nd
shell
special
substitution
vim
Parent
directory 2nd
of all processes
process 2nd 3rd
Parentheses
grouping commands
shell functions
Parse 2nd
Partition
Partner net
PASC
Passive FTP
Passive hub
Passphrase
passwd database
passwd file
passwd utility 2nd 3rd 4th
Password
change 2nd
criteria
defined
Keychain
security
paste utility
PASV FTP. [See Passive FTP]
PATH variable
about
builtin and
find executables using which
inherited
login
Superuser
tcsh
usage
path variable
Path, search
Pathname
absolute 2nd 3rd 4th
completion
defined 2nd
element
expansion 2nd 3rd
last element of
relative 2nd 3rd 4th
special
using
pax utility
pdf filename extension
Performance, system
Period, leading in a filename
Peripheral device. [See Device]
Permission
access
change using chmod
control of
directory
display using ls
execute
read
types of
ACL
execute
file access 2nd
read
setgid
setuid
setuid, set using chmod
Persistent
Philosophy, UNIX 2nd 3rd
Phish, defined
Physical device
PID
$! variable, and
$$ variable 2nd
about
background process, and 2nd
defined
determining using ps
fg
kill, using with 2nd
number 2nd 3rd
temporary file, use in name of
ping utility 2nd
ping6 utility
Pipe
command separator
defined 2nd
end of line, at
filter 2nd
introduction
named 2nd 3rd 4th
noclobber and
standard error, and
symbol
syntax exception
Pipeline. [See Pipe]
Pixel
pkg filename extension
Plaintext
plan file. [See .plan]
plist filename extension
Pluggable Authentication Module. [See PAM]
Plus sign
plutil utility 2nd
Point-to-point link 2nd
Pointer to a file
popd builtin 2nd 3rd
Port
defined
forwarding
privileged
Portability
portmap daemon 2nd 3rd
Portmapper
Position-independent code
Positional parameter 2nd 3rd
POSIX 2nd
Postdecrement operator 2nd
postfix daemon
Postincrement operator 2nd
Postscript, brace expansion
PPID. [See Parent; process]
PPP protocol
pr utility
Preamble, brace expansion
Predecrement operator
Preference file. [See Startup file]
Preincrement operator
Preprocessor directive
#define
#include
defined
macro 2nd
symbolic constant 2nd
Prerequisites, make
Printable character
printcap file 2nd
printenv builtin
Printer
capability database
control characters
lpr and
page break
skip to top of page
top of form
using the
PRINTER variable
Printing
a file. [See Printer; using the]
CUPS
Private address space
private directory
Privilege, least 2nd
Privileged port 2nd
Problems
hung program
login
Procedural language
Procedure
Process
background 2nd 3rd
child 2nd 3rd
defined 2nd 3rd
die 2nd
first
foreground
fork
fork. [See Fork]
ID. [See PID]
init
kill
parent 2nd
parent of all
parent-child relationship
PID, display using ps
search for using ps and grep
sleep 2nd
spawn. [See Fork]
spontaneous
start a
structure
substitution
wake up 2nd 3rd
Processing a command line
procmail utility
profile file 2nd 3rd 4th
Program
badtabs.c
defined
keeping current
stop
structures
tabs.c
terminate
Programmer, applications
Programmer, systems
Programming tools
PROM
Prompt
#
$
%
bash
defined
job control and
login 2nd
PS2
PS3
representation
root
secondary
shell 2nd
Superuser
tcsh 2nd
variable
prompt variable
prompt2 variable
prompt3 variable
Property list files 2nd
Protected memory
Protocol
connection-oriented 2nd
connectionless
datagram-oriented
defined 2nd
DHCP
HTTP
HTTPS
ICMP
IPP
ISO model
network
NNTP
PPP
SLIP
TCP/IP
TELNET 2nd
UDP
protocols file
Proxy
defined 2nd
gateway 2nd
server 2nd
ps filename extension
ps utility 2nd 3rd 4th 5th
PS1 variable 2nd
PS2 variable 2nd
PS3 variable
PS4 variable
pub directory
Public-key encryption, OpenSSH
Puma
pushd builtin 2nd 3rd
pushdsilent variable
pushdtohome variable
putchar C macro
pwd builtin 2nd 3rd
pwd utility
PWD variable
Python


Q

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

quartz-wm window manager
Question mark
Questions, finding answers to
Quick substitution
Quiescent
Quotation mark
double 2nd 3rd 4th
removal
single 2nd 3rd
single versus double 2nd
usage message
Quoting
characters 2nd
defined
shell variables
special characters
variables
whitespace


R

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

Radio button
RAM
defined
disk
Random access memory. [See RAM]
Random bytes, generating
random file
Random number generator
RANDOM variable
RAS
rc file
rc.local file
rc.netboot file
rcp utility 2nd
RDF
Read
access 2nd
user input
read builtin 2nd 3rd 4th 5th
read() system call 2nd
Readline Library
about 2nd
completion
readnews utility
readonly builtin 2nd 3rd 4th
Readonly memory. [See ROM]
Readonly variable
reboot utility
rec.skiing newsgroup
Recursion
defined
example
infinite, alias 2nd
Recursive plunge. [See Recursion; infinite, alias]
Redirect
data fork
operators, bash
output
standard
error 2nd 3rd 4th
input 2nd
output 2nd 3rd 4th 5th
output and error
output of background job
output using tee
output, and append
Redirection 2nd 3rd 4th
Reentrant code 2nd
Reexecuting commands 2nd
Refresh screen
Regular character
Regular expression
\(...\) bracket expression
ampersand 2nd
anchor
asterisk
awk
bracket
bracketing
caret
character class
defined
delimiter
dollar sign
empty
extended 2nd
full
grep
list operator
longest match
period
quoted digit
quoting parentheses
quoting special characters
replacement string
rules of use
simple string
special character
special character, quoting
square bracket
summary
vim
rehash builtin
Relational operator
Relative pathname 2nd 3rd 4th
release command (cvs)
Release, CVS
Religious statue, miniature. [See Icon]
Remainder operator
Remote
Access Server. [See RAS]
filesystem
procedure call. [See RPC]
Remove
directory using rmdir
file using rm
link using rm
variable
remove command
Rename
directory using mv
file using mv 2nd
rename shell script
renice utility
Repair filesystem
repeat builtin
Repeating a command
Replacement string 2nd 3rd 4th
REPLY variable 2nd
Request for Comments. [See RFC]
Reserved port. [See Privileged port]
reset utility
Resolver 2nd
Resource Description Framework. [See RDF]
Resource fork 2nd 3rd
Restore
Return code. [See Exit; status]
RETURN key 2nd 3rd 4th
Reverse single quotation mark. [See Back tick]
rexecd daemon
RFC
RFS
Ritchie, Dennis
rlogin utility 2nd
rm utility 2nd 3rd 4th 5th 6th
rmdir utility 2nd
rmstar variable
rn utility
Roam
ROM
Root
directory 2nd 3rd 4th
filesystem
login
root directory
root login
root user. [See Superuser]
ROT13
Rotate file
Router, network 2nd 3rd
RPC 2nd
rpc file 2nd
rpcinfo utility
RSA encryption
rsh utility 2nd 3rd
rtag command (cvs)
Run
background command
defined
program
shell script


S

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

S filename extension
s filename extension 2nd 3rd 4th
Safari browser 2nd 3rd
safedit shell script
Samba
defined
NBT
NetBIOS
share
SMB
Windows share
WINS
savehist variable 2nd
sbin directory
sched builtin
Scheduling jobs
Schema
sci.med newsgroup
scp utility 2nd
Screen
file
refresh
script utility
Script, shell. [See Shell, script]
Scroll
Scrollbar
sdiff utility
Search
engine
database
defined
find a
index
Web crawler
keyword using apropos
path
string using grep
string using vi 2nd
Secondary prompt
secure file 2nd
securetty file
Security
access permission 2nd
ACL 2nd
authentication
back door
checksum
cipher
ciphertext
cleartext
cookie
cracker
cryptography
cypher. [See Security; cipher]
DDoS attack
DoS attack
finger utility
firewall
host based trust
host, trusted
hosts.equiv file
Internet, root access
IP spoofing
IPng
IPv6
Kerberos
kill
locktty script
login account
Mac OS X features
PAM
password 2nd
PATH and Superuser
PATH variable
rexecd daemon
root access, Internet
RSA
setgid
setuid 2nd
spoofing
ssh. [See ssh]
su utility
sudo utility
Superuser and PATH
telnet
Trojan horse 2nd
trust
trusted host
virtual private network
virus
VPN
wiping a file
worm 2nd
xinetd daemon. [See xinetd daemon]
sed utility 2nd
Segment, network
select control structure
self-assigned IP address
sendmail utility
sendmail, masquerade
Separating commands
Server
defined
mail list 2nd
Message Block Protocol. [See Samba]
name 2nd
process
proxy
setting up
Service, directory
Services
daemons providing
Internet
network
nsswitch.conf file
services database
services file 2nd 3rd
Session
defined
failsafe
set builtin 2nd 3rd 4th 5th 6th 7th 8th
Set group ID. [See Setgid]
Set user ID. [See Setuid]
setenv builtin 2nd 3rd
SetFile utility
Setgid
about
defined
display using ls
permission
root, files belonging to the group 2nd
Setuid
about 2nd
at
crontab
defined
finding files using find
grant privileges
ls display of
root, files owned by 2nd
Sexillion
sh filename extension
sh Shell 2nd 3rd
sh utility
shar shell script
Share
share directory
Shared
library, creating
library, gcc
library, using
network topology
object, filename extension
Shell
about
archive
arithmetic (bash)
calling program, name of
command
grouping 2nd
interpreter
separation
substitution 2nd
comment
comparing strings
control structure
break
case 2nd
continue
do 2nd 3rd 4th
done 2nd 3rd 4th
elif
else
esac
fi 2nd
for 2nd 3rd
for...in 2nd
goto
if 2nd 3rd 4th
if...then
if...then...elif 2nd
if...then...else 2nd
in
switch
then 2nd 3rd
until
while 2nd 3rd
defined
environment variable 2nd
exit from
features
function 2nd
job control
keyword
break
breaksw
case
continue
default
else
endif
endsw
switch
then
variable
while
login 2nd 3rd 4th 5th
name of the calling program
options. [See Shell; features]
parameter
about
positional
special
prompt 2nd 3rd 4th
readonly variable
sh 2nd 3rd
sleep
strings, comparing
user-created variable
variable. [See Shell variable]
Shell script
# comment
#! shell to use
/dev/tty for a terminal
about 2nd
addbanner
bash
birthday
bundle
chkargs 2nd
cleanup
command_menu
comment
count
cpdir
create
debug
defined
double quotation marks 2nd
error message 2nd
executing 2nd
find_uid
finder
Here document
indent
infinite loop
invocation
is_ordfile
links
locktty 2nd
makepath
menu
out
PATH usage
quiz
quote in 2nd 3rd 4th
read user input
recursion
rename
running a
safedit
shar
specifying a shell
temporary filename 2nd
usage message 2nd 3rd 4th
user input
whos
whoson
word_count
word_usage
SHELL variable 2nd
Shell variable
$!
$# 2nd 3rd 4th
$#argv
$$ 2nd 3rd 4th
$* 2nd 3rd
$0
$<
$? 2nd
$@ 2nd
argv
autocorrect
autolist 2nd
autologout
BASH_ENV
CDPATH 2nd
cdpath 2nd 3rd
COLUMNS
correct
CVSROOT
cwd
dirstack
dunique
DYLD_LIBRARY_PATH
echo
EDITOR 2nd
ENV
FCEDIT
fignore
filec
gid
histfile 2nd 3rd
HISTFILESIZE
histlit 2nd
history 2nd
HISTSIZE
HOME 2nd 3rd 4th 5th
home
IFS
ignoreeof 2nd
input_line
INPUTRC
keyword
LESS
LINES
listjobs
listlinks
loginsh
LOGNAME
MAIL
mail
MAILCHECK
MAILPATH
MAILTO
MANPATH
naming
nobeep
noclobber 2nd 3rd
noglob
nonomatch
notify
OLDPWD
OPTARG
OPTIND
owd
PAGER 2nd
PATH
about
builtins do not use
example
find executables using which
keyword shell variable
login
security
tcsh
path
positional parameter
PRINTER
prompt
prompt2
prompt3
PS1 2nd
PS2 2nd
PS3
PS4
pushdsilent
pushdtohome
PWD
quoting
RANDOM
readonly
REPLY 2nd
rmstar
savehist 2nd
SHELL 2nd
shell
shlvl
status
switches
tcsh 2nd
tcsh
TERM 2nd 3rd
time
tperiod 2nd
user
verbose
version
VIMINIT
visiblebell
VISUAL
watch
who
shell variable
shells file
shift builtin 2nd 3rd 4th
shlvl variable
Short-circuiting operator
Shortcut. [See Link]
shutdown utility
Signal
defined 2nd
hang up
kill
list of
names 2nd
quit
software termination
TERM
terminal interrupt
Silicon Graphics
Simple filename 2nd 3rd 4th
Single quotation mark 2nd
Single versus double quotation marks
Single-user
about
maintenance
Superuser
system
Size of file, display using ls
Skip to top of page
sleep utility 2nd
Sleep, shell 2nd
Slice. [See Partition]
SLIP protocol
SMB. [See Samba]
smbd daemon
Smiley
SMTP
Snap, window
Sneakernet
Sniff
soc.singles newsgroup
Socket
about
UNIX domain
SOCKS
Soft link. [See Symbolic; link]
Software termination signal
Sort
sort utility 2nd 3rd 4th 5th 6th
source builtin 2nd
Source code management
Source repository
SPACE 2nd
SPACE Boolean operator 2nd
Spam
defined
whois and
Sparse file 2nd
Spawn. [See Fork]
Special
aliases
character
*
?
[]
defined 2nd
filename generation
Here document
pathname expansion
quoting 2nd
regular expressions
standard input
file 2nd
block
character
parameters, shell 2nd
pathname
Speed, Internet
Spinner
splint utility
split utility
Splitting, word
Spontaneous process
Spoofing, IP
Spool
spool directory 2nd 3rd
SQL
Square bracket
defined
test
ssh utility 2nd 3rd 4th
sshd daemon 2nd
Stack
defined
directory, manipulation 2nd
LIFO
PAM
Stallman, Richard 2nd 3rd 4th 5th 6th
Stand-alone computer
Standard
directories and files
error 2nd 3rd
defined 2nd
exec
file descriptor 2nd
redirect
shell script
trap
input
defined 2nd
exec
file descriptor 2nd
pipe (|)
redirect
special character
output
append
defined 2nd
exec
file descriptor 2nd
pipe (|)
redirect 2nd 3rd
Standards, option handling
Start
emacs
vim
Startup disk 2nd
Startup file
.bash_login file
.bash_logout file
.bash_profile file 2nd
.bashrc file 2nd 3rd
.cshrc file 2nd
.emacs 2nd 3rd 4th
.history file 2nd
.inputrc file
.login file 2nd 3rd
.logout file 2nd
.profile file 2nd
.tcshrc file 2nd
.vimrc file 2nd
/etc
bashrc file 2nd
csh.cshrc file
csh.login file
csh.logout file
cshrc
login
profile file 2nd 3rd
about
bash
BASH_ENV variable
defined
ENV variable
listing
source to execute
tcsh
vim
VIMINIT variable
stat utility
stat() system call 2nd 3rd
Static IP address
Statically linked library
Status
exit
line
line, vim
status variable
stdio.h C header file
Steele, Guy
Sticky bit 2nd
Stop a job
Stop a program
stop builtin
Streaming tape
Streams. [See Connection-oriented protocol]
String
comparing
defined
double quotation marks
finding with grep
pattern matching (bash)
variable
variable, array
strings utility
Stroustrup, Bjarne
Structure, data
Structured Query Language. [See SQL]
stty utility 2nd
Stylesheet. [See CSS]
su utility 2nd
Subdirectory 2nd
Subnet
about
address
defined 2nd
mask 2nd
number
specifying
Subpixel hinting
Subroutine. [See Procedure]
Subshell 2nd 3rd 4th
Substitution [See also Expansion]
command 2nd
parameter
sudoers file
Sun Microsystems
Superblock
Superserver
Superserver daemon. [See xinetd daemon]
Superuser
about
becoming
defined 2nd
explained
multiuser mode
password, changing a user's
PATH, and security
powers
priority 2nd
privileges
prompt
setuid
single-user mode
sudo utility
suspend builtin
Suspend key 2nd 3rd 4th 5th 6th 7th 8th
SVID. [See System; V Interface Definition]
Swap
defined
file, vim
space
switch control structure
Switch. [See Network; switch]
Switching hub
Symbol table
Symbolic
constant
debugger
hostname
link 2nd 3rd 4th 5th 6th
creating using ln
defined 2nd
deleting using rm
tcsh
symlink. [See Symbolic; link]
sync utility
Syntax, command line
sysctl utility 2nd
syslog utility
syslog.conf file
syslogd daemon 2nd
System
administration
client, specifying
configuration file rules
cron utility
dmesg utility 2nd
filesystem repair
halt utility 2nd
host, specifying
kill builtin 2nd
killall utility
more information
multiuser mode
multiuser/graphics mode
newfs utility
reboot utility
rpcinfo utility
setuid files, finding
shutdown utility
single-user mode
startup scripts
su utility
subnet, specifying
sync utility
system does not boot
Trojan horse
umask builtin
uname utility
xinetd daemon
administrator
about
defined
powers 2nd
responsibilities
Superuser
boot
bring down
bring up
call
about 2nd 3rd
bad, trapping
C, from
chmod()
chown()
close()
creat()
defined 2nd
exec() 2nd
exit()
filesystem operations
fork
fork() 2nd 3rd 4th
getpid()
getppid()
kill()
lseek()
manual section
nice()
open()
process control
read() 2nd
stat() 2nd 3rd
tracing with ktrace
unlink()
wait()
write() 2nd 3rd 4th
console 2nd
crash
dataless
does not boot
load average
maintenance
mode
operation
performance
Preferences application, panes
programmer
shutdown
single-user
V Interface Definition
V UNIX
well-maintained
System directory
system services database
system.log file


T

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

T-1 line
T-3 line
TAB key
Table, hash
tabs file
tabs.c program
tail utility 2nd 3rd 4th
talk utility 2nd
talk.politics newsgroup
talkd daemon
tar file
tar utility 2nd 3rd
tar.bz2 filename extension 2nd
tar.gz filename extension 2nd
tar.Z filename extension 2nd
Target file, make
tbz filename extension
tbz2 filename extension
TCP
TCP/IP 2nd
tcsh
alias
arithmetic expression
array of variables 2nd
assignment statement
builtins
control structure
directory stack manipulation 2nd
environment
event number
expression operator
features shared with bash
filename substitution
globbing
hash mechanism 2nd
history mechanism
job control 2nd 3rd 4th
leaving
nice builtin
nohup builtin
numeric variable 2nd
positional parameters
prompt 2nd
quoting variables
redirect, separate stdout from stderr
shell variable
special aliases
startup file 2nd
string variables
switch variables
variable array
variables
word completion
tcsh variable
tee utility 2nd
Teletypewriter 2nd
TELNET protocol 2nd
telnet utility 2nd 3rd
telnetd daemon
Temporary file 2nd
Tera-
TERM signal
TERM variable 2nd 3rd
Termcap
termcap file
Terminal
ASCII
character-based
console
defined
emulator, telnet
file
interrupt signal
name
ansi
vt100
vt102
vt220
xterm
specifying
standard input
standard output
X
Terminal utility 2nd
Terminate a program
Terminfo
terminfo database
terminfo directory
Ternary operator 2nd
test builtin 2nd 3rd 4th 5th 6th 7th 8th 9th
test utility 2nd
Text
box
console 2nd
echo
file
tgz filename extension
Theme
then control structure 2nd 3rd 4th
Thicknet 2nd
Thinnet 2nd
Thompson, Ken 2nd
Thread
about
reentrant code
safe. [See Reentrant code]
Thumb
tif filename extension
tiff filename extension 2nd
Tiger
Tilde expansion 2nd 3rd
Tiled windows
time builtin 2nd 3rd
Time to live. [See TTL]
time variable
tin utility
tmp directory 2nd
Toggle
Token
Token ring network
Tooltip
Top of form
top utility
Top-down design
touch utility 2nd
tperiod variable 2nd
tput builtin
tr utility 2nd 3rd 4th 5th
traceroute utility
traceroute6 utility
Transfer rate, network
Transient window
Transmission Control Protocol. [See TCP]
Transmission Control Protocol/Internet Protocol. [See TCP/IP]
trap builtin 2nd 3rd
Tree structure
Trojan horse 2nd 3rd
true utility
Trusted host
tset utility
TTL
tty file
tty utility 2nd
TTY. [See Teletypewriter]
Tunneling
Tutorial
FTP
Getting Started with emacs
Using vim to create and edit a file
Twisted pair cable
txt filename extension 2nd
type builtin 2nd
Type of file, display using ls
Typeface conventions
typescript file
typeset builtin 2nd
Typo, correcting
tz filename extension


U

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

U.S. Library of Congress
UDP 2nd
UDP/IP
UFS filesystem. [See Filesystem; UFS]
UID
chown
defined
effective
find
ulimit builtin
umask builtin 2nd 3rd 4th
unalias builtin 2nd 3rd 4th
uname utility
uncompress utility
Undecillion
Undeclared variable
unhash builtin
Unicast packet 2nd
Unicast vs. broadcast
uniq utility 2nd
Unique filename 2nd
Universal Coordinated Time. [See UTC]
University of Illinois
UNIX
Berkeley. [See Berkeley; UNIX]
Bourne Shell
domain socket
philosophy
System V 2nd
System V Interface Definition. [See System; V Interface Definition]
unlimit builtin
unlink() system call
Unmanaged window
Unpack archive file using tar
unset builtin 2nd 3rd 4th 5th 6th
unsetenv builtin 2nd
Unshielded twisted pair. [See UTP]
until control structure
unzip utility
update command (cvs)
uptime utility 2nd
Uptime, display using w
URI
URL 2nd
Usage message 2nd 3rd 4th 5th 6th
Usenet 2nd
User
authentication
communication, network
created variable 2nd
Datagram Protocol. [See UDP]
finger
ID. [See UID]
interface 2nd
mode
private groups
root. [See Superuser]
variables, special forms
w
who
user variable
Username
Users directory
Userspace
UTC 2nd
Utility
[]. [See test]
apropos 2nd 3rd 4th
at
atq
atrm
autoconf
awk 2nd 3rd
basename 2nd 3rd
batch
bisonu
builtin, versus
bunzip2 2nd
bzcat 2nd
bzip2 2nd 3rd
bzip2recover 2nd
cal
cat 2nd 3rd 4th 5th 6th 7th
chflags
chgrp
chmod 2nd 3rd 4th
chown
chsh 2nd
clear
cmp
comm
compress 2nd 3rd 4th
configure
cp 2nd 3rd 4th
cpio
cron 2nd
crontab
cut 2nd 3rd
cvs
date 2nd 3rd 4th
dd
ddd
defined
df
diff 2nd
diff3
dig
disktool
diskutil
ditto
dmesg 2nd
du 2nd
dyld 2nd
echo
ed 2nd
egrep 2nd 3rd
elvis
emacs
ex 2nd
expr
fgrep 2nd
file 2nd 3rd
find 2nd 3rd 4th
finding using type
finding using which
finger 2nd 3rd 4th 5th 6th
fink
flex
fmt
fsaclctl
fsck 2nd
ftp 2nd
g++
gcc 2nd 3rd
gcc (GNU)
gdb 2nd 3rd
GetFileInfo
getty 2nd
gopher
gprof
grep 2nd 3rd 4th 5th 6th 7th 8th 9th
groups
gunzip 2nd
gzip 2nd 3rd
halt 2nd
hdiutil
head 2nd
host
hostname 2nd
id
info 2nd 3rd
kdump
kill 2nd
killall 2nd
ktrace
launchctl 2nd
ldd
less 2nd 3rd 4th 5th 6th
lint
ln 2nd 3rd 4th
lock
loginwindow
lpq 2nd
lpr 2nd 3rd 4th
lprm 2nd
lpstat
ls 2nd 3rd 4th 5th 6th 7th 8th
make 2nd 3rd 4th 5th 6th 7th
man 2nd 3rd 4th
mesg
mkdir 2nd 3rd
mkfifo
more
mount
mv 2nd 3rd 4th
names, typeface
netstat
network 2nd
newfs
newfs_hfs
newfs_msdos
nice
nidump 2nd 3rd
nireport
nn
nohup 2nd
nvi
od 2nd
open
option
otool 2nd 3rd
passwd 2nd 3rd 4th
paste
pax
ping 2nd 3rd 4th
plutil 2nd
pr
procmail
ps 2nd 3rd 4th 5th
pwd
rcp 2nd
readnews
reboot
renice
reset
rlogin 2nd
rm 2nd 3rd 4th 5th 6th
rmdir 2nd
rn
rpcinfo
rsh 2nd 3rd
scp 2nd
script
sdiff
sed 2nd
sendmail
SetFile
sh
shutdown
sleep 2nd
sort 2nd 3rd 4th 5th 6th
splint
split
ssh 2nd 3rd 4th
stat
strings
stty 2nd
su 2nd
sync
sysctl 2nd
syslog
tail 2nd 3rd 4th
talk 2nd
tar 2nd 3rd
tee 2nd
telnet 2nd 3rd
Terminal 2nd
test 2nd 3rd 4th 5th 6th
tin
TkCVS
top
touch 2nd
tr 2nd 3rd 4th 5th
traceroute 2nd 3rd
true
tset
tty 2nd
typeset
uname
uncompress
uniq 2nd
unzip
uptime 2nd
uucp
view
vim
vimtutor
w 2nd
watch
wc 2nd 3rd
whatis
which 2nd
who 2nd 3rd 4th 5th 6th
whois
write 2nd 3rd
xargs 2nd
xrn
xvnews
zcat 2nd 3rd
zdiff
zip
zless
UTP cable
uucp utility


V

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

var directory
Variable
about
array
array of numeric
braces
C Shell
completion
default value, assign
defined
display error message
environment 2nd
expansion 2nd
exported
global 2nd
keyword
local 2nd 3rd
modifiers
name, tcsh
naming 2nd
numeric
quoting
readonly
remove
shell
substitute default value
substitution 2nd
tcsh
undeclared
user created 2nd
Verbose boot
verbose variable
version variable
vi [See also vim]
bash command line editor
clones
emacs, versus
simulation in emacs
view utility
Viewport. [See Workspace]
vim
$
( key
) key
*
. command 2nd
. special character
.bash_profile file
.login file
.vimrc startup file 2nd
/ key
key
:
? key
@ symbol
[]
\>
^
A command
a command
abnormal termination
about
active links
address
advanced commands
any character indicator
Append command 2nd
automatic matching
autowrite
B key
b key
beginning-of-line indicator
beginning-of-word indicator
blank delimited word, about
break a line in two
buffer
General-Purpose 2nd
Named
C command
c command
Case command
case sensitivity 2nd
cc command
Change command 2nd
change text
character class definition
character, about
command
Command mode 2nd
compatibility 2nd
CONTROL-B key
CONTROL-D key
CONTROL-F key
CONTROL-G key 2nd
CONTROL-T key
CONTROL-U key
CONTROL-V key
CONTROL-W key
copy text
correct a mistake 2nd 3rd
correct text
crash recovery
create a file
current character
cursor movement. [See vim; move the cursor]
d command 2nd
dd command 2nd
Delete character command
delete characters
Delete command 2nd
delete text 2nd
DOWN ARROW key
edit a file
edit another file
end a session 2nd
end-of-line indicator
end-of-word indicator
enter text
erase key
ESCAPE key
exit from
exit, emergency 2nd
f command
File command
file size
formatting using fmt
G key
General-Purpose buffer 2nd
getting started
Goto command
H key
h key
history
home page
I command
i command
incremental search
indention 2nd
Input mode 2nd 3rd 4th
Insert command
insert text 2nd 3rd
invisible characters
J command
j key
Join command
k key
L key
l key
Last Line mode 2nd 3rd 4th 5th
LEFT ARROW key
line
about
kill key
length
number 2nd
wrap
links, active
m command
M key
marker
matching, automatic
mistake, correct
modes
move text
move the cursor
about 2nd
by characters
by lines
by paragraphs
by sentences
by words
to a specific character
within the screen
N command
n command
Named buffer
Normal mode. [See vim; Command mode]
number, line
O command
o command
open a line
Open command 2nd
overwrite text
P command
p command
page break
PAGE DOWN key
PAGE UP key
paragraph, about
parameter
autoindent
autowrite
compatible
flash
ignorecase
incsearch
list
magic
noautoindent
noautowrite
nocompatible
noflash
noignorecase
noincsearch
nolist
nomagic
nonumber
noreport
noshiftwidth
noshowmatch
noshowmode
nowrapscan
number
report
set
shell 2nd
shiftwidth
showmatch
showmode
wrapmargin
wrapscan
Put commands
q command
quit 2nd
Quit command
Quote command
R command
r command 2nd
Read command
recover text
redraw the screen
regular expression
Repeat command
Repeat Factor 2nd 3rd
replace
Replace command 2nd
replace string
replace text
replacement string
report on changes
RETURN key 2nd
RIGHT ARROW key
S command
s command 2nd
safedit script
screen, about
scroll
search
search and replace
search command
search string 2nd
search, incremental
sentence, about
set
marker
parameter
shell
showmatch
SPACE bar
special characters 2nd 3rd 4th
start over
starting
startup file
Status command
status line
Substitute command 2nd
substitute text
substitution
summary
terminal specification
tilde symbol
transpose command
U command
u command 2nd 3rd
undo changes 2nd
Undo command 2nd
Unit of Measure 2nd 3rd
UP ARROW key
vi compatibility
view different parts of the Work buffer
view utility
VIMINIT variable
w command 2nd
W key
w key
word, about
Work buffer 2nd 3rd
Write address
Write command 2nd
x command 2nd
y command
Yank command
yy command
ZZ command 2nd
{ key
} key
~ command
~ symbol
VIMINIT variable
vimtutor utility
Virtual
console
private network. [See VPN]
Virus
visiblebell variable
Visiting a file, emacs 2nd
VISUAL variable
Vixie cron
VLAN
Volume. [See Partition]
Volumes directory
VPN 2nd
vt100 terminal
vt102 terminal
vt220 terminal


W

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

w utility 2nd
W2K
W3. [See World Wide Web]
W3C
wait builtin 2nd
wait() system call
Wake up, process
WAN 2nd 3rd 4th 5th
WAP
watch utility
watch variable
wc utility 2nd 3rd
Web [See also World Wide Web]
crawler
weekly file
whatis utility
wheel group
where builtin
which builtin 2nd
which utility 2nd
while control structure 2nd 3rd 4th 5th
Whitespace
command line
defined 2nd
quoting
who am i
who utility 2nd 3rd 4th 5th 6th
who variable
whois database
whois utility
whos shell script
whoson shell script
Wi-Fi
Widget
Wildcard
Wildcard. [See also Metacharacter]
Window
cascading
defined
ignored
manager
about
quartz-wm
minimize
scrollbar
share. [See Samba, share]
snap
thumb
tiled
transient
unmanaged
Windows
filename limitation
privileged port
WINS
Wiping a file
Wire. [See Cable]
Word
completion
defined 2nd 3rd 4th
deleting
designator
erase key 2nd
parse a command line
splitting (bash)
word_count shell script
word_usage shell script
Work buffer, vim 2nd 3rd 4th
Working directory
change using cd
defined 2nd
execute a file in 2nd
PATH
relative pathnames and
significance of
versus home directory
Workstation
World Wide Web
about
Berners-Lee, Tim
CERN
Enquire
HTML
hypermedia
hypertext
link, hypertext
Mosaic browser
name
origins
overview
search engine
URL
Web crawler
Worm 2nd
Write access
write utility 2nd 3rd
write() system call 2nd 3rd 4th
wtmp file
WWW. [See World Wide Web]
wysiwyg


X

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

X Window System
about
defined
emacs
X11 directory
X11R6 directory
X11R6 directory
xargs utility 2nd
Xcode Tools
XDMCP
xDSL
xinetd daemon 2nd 3rd 4th 5th
xinetd.conf file
XML 2nd
xrn utility
XSM
xterm terminal name
xvnews utility


Y

Previous Page
Next Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

y filename extension


Z

Previous Page

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

Z filename extension 2nd
Z Shell
zcat utility 2nd 3rd
zdiff utility
zero file
zip utility
zless utility
zsh shell
Zulu time. [See UTC]